Deployment
Local Development Environment
MinIO (S3-compatible local storage)
Docker Compose setup in infra/docker-compose.yml:
just infra-up
This starts MinIO on localhost:9000 (API) and localhost:9001 (console), plus catalog services depending on active profiles. See INFRASTRUCTURE.md for the full service list.
Credentials: minioadmin / minioadmin
Seeding Test Data
just seed-all # Seed all catalog types (REST, SQL, HMS)
just seed-rest # Lakekeeper REST catalog only
just seed-sql # SQLite SQL catalog only
just seed-hms # Hive Metastore only
See INFRASTRUCTURE.md for seed script details.
AWS Infrastructure (Terraform)
Terraform config in infra/terraform/main.tf.
just tf-init # Initialize backend
just tf-plan # Preview changes
just tf-apply # Apply changes
just tf-destroy # Tear down (careful!)
Provisions:
- S3 bucket for test data, Athena results, and Spark scripts
- Glue database for table registration
- Athena workgroup (engine v3) for query verification
- Lake Formation database-level permissions for IAM_ALLOWED_PRINCIPALS
- Glue ETL job definition (
iceberg-migration-verify, Glue 5.1) - IAM role for the Glue ETL job
- EMR Serverless application (
iceberg-migration-test, EMR 7.12.0) - IAM role for EMR Serverless jobs
- Test Lambda (ECR repo, function-from-image, IAM role, log group) —
infra/terraform/lambda.tf
After tf-apply, copy the outputs (emr_application_id, emr_job_role_arn, glue_job_name) into .env.
AWS Lambda Entry Point
The Lambda is the second Entry point alongside the CLI, invoked synchronously by Airflow over a PrivateLink endpoint. For the event/response contract, IAM permissions, VPC/endpoint prerequisites, image build/push, and an example payload for the client’s platform team, see LAMBDA.md. Build the image:
just lambda-build # docker build --platform linux/amd64 -t iceberg-migrate-lambda:latest .
Verifying the Integration
With infrastructure provisioned and all catalogs seeded:
just test-integration # All engines (Athena + Glue ETL + EMR Serverless)
just test-athena # Athena only
just test-glue # Glue ETL only
just test-emr # EMR Serverless only
Lambda Release Gate (no-egress sandbox)
To run the complete release gate — provision a throwaway no-egress sandbox VPC, run the differential-equivalence test (Lambda artifacts == CLI artifacts), then tear down the sandbox:
just test-lambda-release # provision → test → destroy (teardown guaranteed on failure)
For iterative debugging (apply once, run the test repeatedly):
just tf-sandbox-apply # provision sandbox once
just lambda-push-sandbox # push image to sandbox ECR
SANDBOX_LAMBDA_FUNCTION_NAME=iceberg-migration-sandbox \
uv run pytest tests/integration/test_lambda_invoke.py -m integration -v # repeat as needed
just tf-sandbox-destroy # destroy when done
The sandbox stack lives in infra/terraform/sandbox/ with separate Terraform
state; destroying it never affects the main stack resources (S3 bucket, Glue
database, Athena workgroup).
Running the Tool
From source (development):
uv run iceberg-migrate migrate \
--table-location s3://bucket/warehouse/db/table \
--source-prefix s3a://old-bucket/warehouse \
--dest-prefix s3://bucket/warehouse
Installed:
pip install .
iceberg-migrate migrate \
--table-location s3://bucket/warehouse/db/table \
--source-prefix s3a://old-bucket/warehouse \
--dest-prefix s3://bucket/warehouse
AWS Credentials
The tool uses the default boto3 credential chain:
- Environment variables (
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY) - AWS config files (
~/.aws/credentials,~/.aws/config) - IAM role (EC2/ECS/Lambda)
Required permissions:
s3:GetObjecton source metadata filess3:PutObjecton_migrated/prefixglue:CreateTable,glue:GetTable,glue:UpdateTableon target database
Source: docs/DEPLOYMENT.md on GitHub ↗