Development
Prerequisites
- Python >= 3.10 (3.12 recommended)
- uv — package manager
- just — task runner
- Docker (for local catalog tests and MinIO)
- act — local CI runner (optional)
For integration tests (optional)
- AWS CLI configured (profile set in
.env) - Terraform >= 1.0
Setup
# Clone and install
git clone <repo-url>
cd iceberg-metadata-migration
uv sync --dev
# Install pre-commit hooks
uv run prek install
# Install just (macOS)
brew install just
Quick Reference
just --list # Show all available commands
just test # Unit tests
just lint # Lint check
just typecheck # Type check
just check # All checks (lint + types + tests)
Local Catalog Testing
Start Docker services and seed test data:
# All catalogs at once
just infra-up && just seed-all
# Or individually
just seed-rest # Lakekeeper REST catalog
just seed-sql # SQLite SQL catalog (MinIO only)
just seed-hms # Hive Metastore
# Run tests
just test-local # All catalog tests
just test-rest # Just REST
See INFRASTRUCTURE.md for Docker compose details.
Integration Testing (AWS)
Requires AWS credentials and Terraform:
# One-time: provision infra
just tf-init
just tf-apply
# Run integration tests
just test-integration # All engines (Athena + Glue ETL + EMR Serverless)
just test-athena # Athena only
just test-glue # Glue ETL only
just test-emr # EMR Serverless only
Additional .env vars required for multi-engine tests: EMR_APPLICATION_ID, EMR_JOB_ROLE_ARN, GLUE_JOB_NAME.
See TESTING.md for full test strategy.
Daily Workflow
Run tests
just test # Unit tests (fast)
just check # Full check: lint + types + tests
Lint and format
just lint # Check
just lint-fix # Auto-fix
Type check
just typecheck
Run full CI locally
act -j check
Pre-commit Hooks (prek)
Configured in prek.toml. Runs on every commit:
- ruff — auto-fix lint issues + format
- basedpyright — type checking
- pytest — fail-fast test run
Install hooks:
uv run prek install
CI Pipeline
GitHub Actions workflow in .github/workflows/ci.yml. Runs on push to master and on PRs.
Same checks as pre-commit: ruff -> basedpyright -> pytest.
Adding Dependencies
# Runtime dependency
uv add <package>
# Dev dependency
uv add --group dev <package>
Project Structure
See ARCHITECTURE.md for module layout and design decisions.
Source: docs/DEVELOPMENT.md on GitHub ↗