Architecture
Module Layout
src/iceberg_migrate/
├── cli.py # Typer CLI entry point — orchestrates the full pipeline
├── models.py # Pydantic models: IcebergMetadataGraph, ManifestListFile, ManifestFile
├── s3.py # S3 utilities: URI parsing, object fetch
├── discovery/
│ ├── compression.py # Extension-based decompression + suffix stripping
│ ├── locator.py # Find latest metadata.json in S3
│ └── reader.py # Load metadata graph from S3 (metadata.json → Avro files)
├── rewrite/
│ ├── config.py # RewriteConfig: source/destination prefix pair
│ ├── engine.py # RewriteEngine: orchestrates rewriting + _migrated/ key remapping
│ ├── metadata_rewriter.py # Rewrite path fields in metadata.json dict
│ ├── avro_rewriter.py # Rewrite paths in manifest list/manifest Avro records
│ └── graph_loader.py # Load ALL snapshots' files for time-travel safe rewriting
├── validation/
│ └── validator.py # Pre-write validation: structural, residual prefix scan, counts
├── writer/
│ └── s3_writer.py # Bottom-up S3 writer (manifests → manifest lists → metadata)
├── catalog/
│ ├── base.py # CatalogRegistrar protocol + CatalogConfig
│ ├── registry.py # Catalog adapter registry (type string → adapter)
│ ├── glue_registrar.py # AWS Glue adapter: register/update table
│ └── rest_registrar.py # REST catalog adapter (Polaris, LakeKeeper, etc.)
└── output/
└── formatter.py # Human (rich) and JSON output formatting
Data Flow
S3 (original metadata)
│
▼
Discovery (locator.py → compression.py → reader.py)
│ Finds latest metadata.json, decompresses if needed, parses JSON
│ Loads Avro manifest lists + manifests; captures codec from each Avro header
▼
IcebergMetadataGraph (models.py)
│ In-memory typed representation of the full metadata tree
│ Each ManifestListFile/ManifestFile carries avro_schema + codec
▼
Graph Loader (graph_loader.py)
│ Extends graph to include ALL snapshots' files (not just current)
▼
RewriteEngine (engine.py)
│ 0. Format-version gate: rejects format-version >= 4
│ 1. Rewrites paths via metadata_rewriter.py + avro_rewriter.py
│ 2. Serializes to bytes (orjson for JSON, fastavro for Avro)
│ 3. Preserves Avro codec per file on serialization
│ 4. Remaps S3 keys to _migrated/ paths (strips compression suffix)
▼
RewriteResult
│ Contains rewritten graph + serialized bytes at _migrated/ keys
▼
Validator (validator.py)
│ Structural check, residual prefix scan, manifest count verification
▼
S3 Writer (s3_writer.py)
│ Writes to _migrated/ keys in bottom-up order
▼
Catalog Registrar (glue_registrar.py or rest_registrar.py)
│ Registers table pointing to _migrated/ metadata.json
▼
Output (formatter.py)
Human-readable (rich) or JSON summary
Key Design Decisions
| Decision | Rationale |
|---|---|
| fastavro over apache-avro | 8x faster, no IgnoredLogicalType bugs with Iceberg manifest schemas |
| Direct S3 writes over pyiceberg StaticTable | StaticTable is read-only; cannot write back rewritten metadata |
| Direct Glue API over pyiceberg GlueCatalog | Avoids s3fs/FileIO dependency for registration; sets same Iceberg Parameters |
Non-destructive _migrated/ writes |
Original metadata may be overwritten by sync tools; migrated copy is independent |
| Bottom-up write order | If a write fails mid-way, lower-level files are consistent; metadata pointer updated last |
| All-snapshot rewriting | Time-travel safety: historical snapshots reference old paths too |
| Extension-based decompression dispatch | Lakekeeper writes .gz.metadata.json (hardcoded gzip). Dispatch table in compression.py uses filename extension to select decompressor. Stdlib gzip is sufficient — no cramjam or third-party compression library needed. |
| Avro codec preservation on round-trip | Codec read from Avro container header during discovery, stored on model, passed back to fastavro.writer(). Prevents silently changing deflate to null or vice versa. |
| Format-version gate (v1/v2/v3 only) | RewriteEngine rejects format-version >= 4 before any rewriting. Prevents silent data corruption on unknown metadata structures. |
| Strip .gz extension from migrated metadata keys | Athena uses file extension to determine encoding. A .gz.metadata.json key containing plain JSON triggers GENERIC_INTERNAL_ERROR. Migrated copies always use plain .metadata.json extension. |
Lessons Learned
These were discovered during integration testing against real catalog-produced metadata.
Gzip discovery
Lakekeeper metadata.json is gzip-compressed with a .gz.metadata.json extension. Initial integration tests failed with JSON parse errors until decompression was added. The fix was compression.py with a dispatch table keyed on filename extension.
Athena .gz extension
Writing plain JSON to a .gz.metadata.json S3 key causes Athena GENERIC_INTERNAL_ERROR. Athena infers encoding from the file extension, not file magic bytes. The fix: strip_compression_suffix() in engine.py removes compression extensions from output keys so migrated files always land at plain .metadata.json.
Avro codec mismatch
Reading deflate-compressed Avro (typical from Java catalogs) and writing back with null codec produces valid Avro but breaks query engines that expected the original codec. Fix: capture codec from avro.codec header in reader.py, store it on the ManifestListFile/ManifestFile model, and pass it through to fastavro.writer() in engine.py.
referenced_data_file (v3)
Position and equality delete manifests use data_file.referenced_data_file (field id 143) to point back to the data file being deleted. If this field is not rewritten, the delete references break after migration — the delete file points to a rewritten data path but the reference is still on the old prefix. The fix: avro_rewriter.py rewrites data_file.referenced_data_file alongside data_file.file_path.
No cramjam needed
Stdlib gzip handles all observed compression formats. The extension-based dispatch table in compression.py is designed to add future codecs (zstd, snappy) as needed, but none have been required yet.
Glue 5.1 catalog naming
Glue ETL’s --datalake-formats iceberg flag only provides Iceberg JARs and configures spark_catalog as the Hive metastore bridge. It does NOT create a named Iceberg catalog backed by Glue. To query Iceberg tables registered in Glue, you need explicit --conf entries that define glue_catalog as a named SparkCatalog with GlueCatalog impl and S3FileIO. Tables are then accessed as glue_catalog.{db}.{table}, identical to EMR Serverless. Without this, Spark only sees spark_catalog which uses Hive metastore semantics, not Iceberg.
Regional S3 endpoints (EMR Serverless)
In some AWS regions (e.g., ap-southeast-5 Jakarta), the global S3 endpoint times out for boto3.client("s3").put_object(). EMR Serverless jobs that write results back to S3 must use boto3.client("s3", region_name=aws_region) with the explicit region. The fix: verify_emr.py accepts --aws_region as a CLI argument, passed through from the test fixture.
Lake Formation per-table grants
Glue ETL and EMR Serverless execution roles have IAM policies allowing glue:GetTable, but Lake Formation blocks table access unless an explicit LF grant exists on each table. The Terraform aws_lakeformation_permissions resource only grants database-level access to IAM_ALLOWED_PRINCIPALS. Per-table grants must be applied after each table is registered — the migrated_tables fixture in conftest.py calls lf.grant_permissions() on every migrated table.
PyIceberg overwrite produces 2 snapshots, not 1
table.overwrite(df, overwrite_filter="id <= 2") creates 2 snapshots internally (OVERWRITE + APPEND), not 1. A seed sequence of append → overwrite → append produces 4 snapshots total instead of the expected 3. The read_snapshot_timestamps() helper addresses this by returning named keys {"s1": timestamps[0], "s2": timestamps[2]} that skip the intermediate OVERWRITE snapshot at index 1, abstracting over the exact snapshot count.
Iceberg time-travel timestamp resolution
Iceberg returns the snapshot with timestamp <= query_timestamp. To query a specific snapshot, add +1ms to its timestamp. Athena uses FOR TIMESTAMP AS OF TIMESTAMP '{iso_string}' syntax; Spark (Glue ETL, EMR Serverless) uses spark.read.option("as-of-timestamp", epoch_ms).table(name).
Multi-snapshot seeding Integration test seed tables use 3 operations (append 5 rows → overwrite 2 rows → append 5 rows) producing 4 snapshots. This enables time-travel verification: snapshot 1 = 5 rows / SUM(amount) 801.5, snapshot 2 = 5 rows / SUM(amount) 2387.25, current = 10 rows / SUM(amount) 3478.0. All three query engines (Athena, Glue ETL, EMR Serverless) verify snapshots 1 and 2.
Source: docs/ARCHITECTURE.md on GitHub ↗