Catalog Abstraction
Overview
The catalog layer provides a pluggable interface for registering migrated Iceberg tables with different catalog backends. Currently supports AWS Glue and REST-based catalogs.
Architecture
catalog/
├── base.py # CatalogRegistrar protocol + CatalogConfig + error types
├── registry.py # Maps catalog_type string → adapter instance
├── glue_registrar.py # AWS Glue implementation
└── rest_registrar.py # REST catalog implementation (Polaris, LakeKeeper, etc.)
CatalogRegistrar Protocol
All adapters implement this protocol (defined in base.py):
class CatalogRegistrar(Protocol):
def register_table(
self,
namespace: str,
table: str,
metadata_location: str,
metadata: dict[str, Any] | None = None,
) -> str: # Returns "created" or "updated"
...
def validate_connection(self) -> bool:
...
CatalogConfig
@dataclass
class CatalogConfig:
catalog_type: str # 'glue' or 'rest'
uri: str | None # REST catalog URI
warehouse: str | None # Warehouse identifier
credential: str | None # OAuth2 client credential
token: str | None # Bearer token
region: str | None # AWS region
properties: dict[str, str] # Additional properties
AWS Glue Adapter
Module: catalog/glue_registrar.py
- Uses boto3 Glue client directly (not PyIceberg’s GlueCatalog)
- Sets Iceberg-specific parameters:
table_type=ICEBERG,metadata_location - Idempotent: creates new table or updates existing via optimistic locking
- Derives database/table names from S3 path (Hive-style:
warehouse/<db>/<table>) - Does NOT create databases — requires pre-existing Glue database
REST Catalog Adapter
Module: catalog/rest_registrar.py
- Implements the Iceberg REST Catalog spec
- Supports OAuth2 (
credential) and bearer token (token) auth - Compatible with Polaris, LakeKeeper, and other REST-based catalogs
Adding a New Adapter
- Create
catalog/<name>_registrar.py - Implement the
CatalogRegistrarprotocol - Register in
catalog/registry.py
Example skeleton:
from iceberg_migrate.catalog.base import CatalogConfig, CatalogRegistrar
class MyRegistrar:
def __init__(self, config: CatalogConfig):
self.config = config
def register_table(self, namespace, table, metadata_location, metadata=None):
# Implementation here
return "created"
def validate_connection(self):
return True
Then in registry.py, add the mapping:
"my_catalog": MyRegistrar
Error Types
All defined in base.py:
| Exception | When Raised |
|---|---|
CatalogError |
Base exception for catalog operations |
TableAlreadyExistsError |
Table exists and cannot be overwritten |
NamespaceNotFoundError |
Target namespace/database doesn’t exist |
CatalogUnreachableError |
Catalog endpoint unreachable |
AuthenticationError |
Auth to catalog fails |
Source Catalog Format Reference
The migration tool reads metadata produced by any Iceberg catalog. Different catalogs write metadata in different formats. This table summarizes what the tool handles during discovery and rewrite.
Metadata.json Encoding
| Source Catalog | Encoding | Filename Pattern | Notes |
|---|---|---|---|
| Java-based (Glue, HMS, Nessie, Polaris, Gravitino, Unity) | Plain UTF-8 JSON (default) | {5-digit}-{UUID}.metadata.json |
Optional gzip via write.metadata.compression-codec |
| Java Hadoop/filesystem | Plain UTF-8 JSON (default) | v{N}.metadata.json |
Uses version-hint.text for version tracking |
| Lakekeeper (Rust) | Gzip-compressed JSON (hardcoded) | {5-digit}-{UUID}.gz.metadata.json |
Not configurable per-table |
| PyIceberg (SQL catalog) | Plain UTF-8 JSON (default) | {5-digit}-{UUID}.metadata.json |
Optional gzip via table property |
Avro Manifest Codec
| Source Catalog | avro.codec Header Value |
Notes |
|---|---|---|
| Java-based (all) | deflate |
Default; gzip/deflate level 9. Iceberg calls it “gzip” but Avro spec name is “deflate” |
| Lakekeeper (post Jan 2026) | deflate |
Changed from null to deflate in iceberg-rust PR #1851 |
| Lakekeeper (pre Jan 2026) | null |
Uncompressed Avro containers |
| PyIceberg | deflate |
Maps "gzip" -> "deflate" in Avro header |
The tool preserves the original codec on round-trip: codec is read from the Avro container header during discovery, stored on the ManifestListFile/ManifestFile model, and passed back to fastavro.writer() during serialization.
Format Version Support
| Version | Supported | Key Additions |
|---|---|---|
| 1 | Yes | Original format; no row-level deletes |
| 2 | Yes | Row-level deletes (position + equality delete files) |
| 3 | Yes | Deletion vectors, referenced_data_file field |
| 4+ | Rejected | Not yet released; RewriteEngine raises ValueError |
Source: docs/CATALOG.md on GitHub ↗