AWS Lambda Entry Point
The Lambda is the second Entry point alongside the CLI. Airflow invokes it
synchronously (RequestResponse) over a PrivateLink endpoint with a JSON payload
that mirrors the CLI arguments 1:1 in snake_case. The handler is a thin adapter:
it maps the event to a MigrationParams, calls the shared run_migration()
(ADR-0001 — same function the CLI calls), and returns the summary as a dict. The
pipeline modules are unchanged.
The CLI remains the other Entry point. Both share one orchestration path; see CLI.md for the flag equivalents and DEPLOYMENT.md for the host path.
Event Contract
A JSON object whose keys are the CLI option names in snake_case. Missing a required key makes the handler raise before any work.
| Key | Required | Default | Notes |
|---|---|---|---|
table_location |
yes | — | S3 URI of the Iceberg table root |
source_prefix |
yes | — | The stale Source prefix to Rewrite |
dest_prefix |
yes | — | The Destination prefix to point metadata at |
glue_database |
no | derived from table location | Override Glue database |
glue_table |
no | derived from table location | Override Glue table |
dry_run |
no | false |
Report what would be written; no S3 writes, no Registration |
verbose |
no | false |
Per-file Rewrite counts in logs |
aws_region |
no | env AWS_DEFAULT_REGION |
Region for Glue; falls back to the Lambda environment |
Response and Failure Contract
- Success returns a JSON object with the same shape as the CLI’s
--json(render_json):status: "success", the rewritten-file counts, and the Glue Registration outcome. Lambda serializes the returned dict. - Any failure — partial or fatal — raises (ADR-0002). Lambda reports a
FunctionError. The caller (Airflow) treatsFunctionErroras a failed task and retries. A retry is safe because the tool is non-destructive and idempotent: Migrated metadata is written under_migrated/and originals are never touched. Partial writes are not rolled back; a retry re-runs the whole Invocation.
This collapses the CLI’s 0/1/2 exit codes into return-vs-raise, because an orchestrator only needs task-succeeded vs. task-failed.
Example Invocation Payload
Minimal required:
{
"table_location": "s3://my-bucket/warehouse/analytics/events",
"source_prefix": "s3a://old-bucket/warehouse",
"dest_prefix": "s3://my-bucket/warehouse"
}
Full payload (mirrors the CLI examples in DEPLOYMENT.md):
{
"table_location": "s3://my-bucket/warehouse/analytics/events",
"source_prefix": "hdfs://namenode:8020/warehouse",
"dest_prefix": "s3://my-bucket/warehouse",
"glue_database": "my_analytics",
"glue_table": "user_events",
"dry_run": false,
"verbose": true,
"aws_region": "us-east-1"
}
IAM Permissions (Least Privilege)
Scope the execution role to:
s3:GetObjecton the source metadata files (manifests,metadata.json)s3:ListBucketon the table bucket (metadata discovery)s3:PutObjectunder_migrated/onlyglue:CreateTable,glue:GetTable,glue:UpdateTable,glue:GetDatabaseon the target databaselogs:CreateLogStream,logs:PutLogEventson the function’s log group- The AWS-managed
AWSLambdaVPCAccessExecutionRolepolicy (required for VPC attachment — grants ENI create/describe)
VPC and Endpoint Prerequisites
The production function runs in private subnets with no internet egress (ADR-0004). It reaches AWS over VPC endpoints that already exist in the client VPC and are not created by this repo:
- S3 — Gateway VPC endpoint (the function reads source metadata and writes Migrated metadata)
- Glue — Interface VPC endpoint (for Registration)
- Lambda invoke path — a separate PrivateLink interface endpoint for Airflow → Lambda control plane. This is independent of whether the function is attached to a VPC.
The function code and image are identical regardless of VPC attachment. These endpoints are client-owned; this repo’s Terraform provisions only a test Lambda.
No-Egress Assertion (Structural Only)
The release gate (just test-lambda-release) runs the function inside a
throwaway sandbox VPC provisioned by infra/terraform/sandbox/. The VPC has
no Internet Gateway and no NAT Gateway — no-egress is asserted structurally
by the absence of these resources in the Terraform config, not by a positive
network probe.
Residual risk: a future Terraform change that added an IGW or NAT to the
sandbox stack would restore internet egress, but the gate would still pass
because nothing in the test positively probes the network path. This is an
accepted trade-off. The mitigation is code review of infra/terraform/sandbox/
changes.
Environment Variables
AWS_DEFAULT_REGION— the region fallback whenaws_regionis omitted from the payload.
The function runs under its execution role for AWS credentials — no static access keys are configured.
Image Build and Push
The image is FROM public.ecr.aws/lambda/python:3.12, x86_64, with dependencies
installed from a uv export-generated requirements.txt for reproducibility
against uv.lock (ADR-0003). The Dockerfile at the repo root sets
CMD ["iceberg_migrate.lambda_handler.handler"].
just lambda-build
# equivalent to:
# docker build --platform linux/amd64 -t iceberg-migrate-lambda:latest .
Push to the ECR repo (requires AWS_ACCOUNT_ID and AWS_REGION in .env, and
the ECR repo from just tf-apply):
just lambda-push
Repository Terraform Scope
This repo’s Terraform provisions a test Lambda only (ADR-0004), in
infra/terraform/lambda.tf: an ECR repository, an aws_lambda_function from the
image (x86_64, memory 1024 MB, timeout 300 s — both variables), the
least-privilege IAM role and policy above, a CloudWatch log group, and optional
vpc_config from lambda_subnet_ids + lambda_security_group_id variables. It
does not create the S3/Glue VPC endpoints, the PrivateLink invoke endpoint,
or the production function — production deployment and networking are owned by
the client’s platform team.
Source: docs/LAMBDA.md on GitHub ↗