Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/python-ci-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ jobs:
- name: Install UV
uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
- name: Build docs
run: make docs-build
run: uv run nox -s docs-build
37 changes: 10 additions & 27 deletions .github/workflows/python-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,62 +33,45 @@ concurrency:
jobs:
lint-and-unit-test:
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
max-parallel: 15
fail-fast: true
matrix:
python: ['3.11', '3.12', '3.13', '3.14']
timeout-minutes: 25

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python }}
- name: Install UV
uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
with:
enable-cache: true
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Check uv.lock is up to date
if: matrix.python == '3.11'
run: uv lock --check
- name: Install
run: make install
- name: Run linters
if: matrix.python == '3.11'
run: make lint
run: uv run nox
# The Python version matrix lives in noxfile.py, so nox runs every supported
# version here instead of a job matrix. --parallel runs them concurrently.
- name: Run unit tests with coverage
run: COVERAGE=1 make test
env:
COVERAGE: "1"
run: uv run nox -s tests -j auto
- name: Generate coverage report (85%) # Coverage threshold should only increase over time — never decrease it!
run: COVERAGE_FAIL_UNDER=85 make coverage-report
run: uv run nox -s coverage

windows-unit-test:
runs-on: windows-latest
timeout-minutes: 15
strategy:
fail-fast: true
matrix:
python: ['3.12']
timeout-minutes: 20

steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python }}
- name: Install UV
uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
with:
enable-cache: true
- name: Install
run: uv sync --all-extras --no-extra hive-kerberos
- name: Run unit tests
run: uv run python -m pytest tests/ -m unit --ignore=tests/integration -v -x
run: uv run nox -s tests -p 3.12

cibw-dev-env-smoke-test:
runs-on: ubuntu-slim
Expand Down
18 changes: 9 additions & 9 deletions .github/workflows/python-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,9 @@ jobs:
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
run: make install
run: uv run nox -s install
- name: Run integration tests with coverage
run: COVERAGE=1 make test-integration
run: COVERAGE=1 uv run nox -s integration
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose-integration.yml logs
Expand Down Expand Up @@ -78,9 +78,9 @@ jobs:
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
run: make install
run: uv run nox -s install
- name: Run s3 integration tests with coverage
run: COVERAGE=1 make test-s3
run: COVERAGE=1 uv run nox -s s3
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose.yml logs
Expand Down Expand Up @@ -108,9 +108,9 @@ jobs:
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
run: make install
run: uv run nox -s install
- name: Run adls integration tests with coverage
run: COVERAGE=1 make test-adls
run: COVERAGE=1 uv run nox -s adls
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose-azurite.yml logs
Expand Down Expand Up @@ -138,9 +138,9 @@ jobs:
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
run: make install
run: uv run nox -s install
- name: Run gcs integration tests with coverage
run: COVERAGE=1 make test-gcs
run: COVERAGE=1 uv run nox -s gcs
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose-gcs-server.yml logs
Expand Down Expand Up @@ -173,7 +173,7 @@ jobs:
pattern: coverage-*
merge-multiple: true
- name: Generate coverage report (75%) # Coverage threshold should only increase over time — never decrease it!
run: COVERAGE_FAIL_UNDER=75 make coverage-report
run: COVERAGE_FAIL_UNDER=75 uv run nox -s coverage

python-integration-required:
if: ${{ always() }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-release-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
- name: Install UV
uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
- name: Build docs
run: make docs-build
run: uv run nox -s docs-build
- name: Copy
working-directory: ./mkdocs
run: mv ./site /tmp/site
Expand Down
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,10 @@ htmlcov
pyiceberg/avro/decoder_fast.c
pyiceberg/avro/*.html
pyiceberg/avro/*.so

# nox session environments
.nox/

# rumdl lint cache
.rumdl_cache/
.ruff_cache/
26 changes: 0 additions & 26 deletions .markdownlint.yaml

This file was deleted.

64 changes: 0 additions & 64 deletions .pre-commit-config.yaml

This file was deleted.

24 changes: 13 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ lives under `pyiceberg/`, organized by concern rather than by engine:

### Style & Typing

- Formatting and linting are enforced by `ruff` via `prek` (pre-commit): line length **130**, double-quoted strings, isort with `pyiceberg`/`tests` as first-party. Run `make lint` — ruff autofixes most issues.
- Formatting and linting are enforced by `ruff` and `mypy`, driven by `nox` sessions defined in `noxfile.py`: line length **130**, double-quoted strings, isort with `pyiceberg`/`tests` as first-party. Run `just fix` then `just check` — ruff autofixes most issues.
- Full type annotations are required (`mypy` runs in strict mode: `disallow_untyped_defs`, `no_implicit_optional`, `warn_unused_ignores`). Avoid Any types.
- Docstrings follow the project's pydocstyle config; one-line summaries on public functions. No personal pronouns in comments.
- Use domain-specific exceptions from pyiceberg/exceptions.py for Iceberg-specific error conditions. For invalid arguments/values, ValueError is acceptable. Don't raise bare Exception.
Expand Down Expand Up @@ -76,15 +76,17 @@ lives under `pyiceberg/`, organized by concern rather than by engine:

## Commands

- **Install / set up dev env:** `make install` (installs `uv`, syncs all extras, builds Cython, installs pre-commit hooks)
- **Run unit tests:** `make test`
- **Run a subset:** `make test PYTEST_ARGS="-v -k <pattern>"`
- **Integration tests (Spark/Docker):** `make test-integration` (rebuild with `make test-integration-rebuild`)
- **Cloud storage suites:** `make test-s3` / `make test-adls` / `make test-gcs`
- **Lint & format:** `make lint`
- **Docs preview / build:** `make docs-serve` / `make docs-build`
- **Clean build artifacts:** `make clean`
- **Use a specific Python:** prefix with `PYTHON=3.12`, e.g. `PYTHON=3.12 make install`
- **Install / set up dev env:** `just install` (installs `uv`, syncs all extras, builds Cython)
- **Run unit tests:** `just test`
- **Run a subset:** `just test -v -k <pattern>`
- **Integration tests (Spark/Docker):** `just test-integration` (rebuild with `just test-integration-rebuild`)
- **Cloud storage suites:** `just test-s3` / `just test-adls` / `just test-gcs`
- **Every CI gate (format, lint, types, hygiene):** `just check`
- **Autofix format and lint:** `just fix`
- **Docs preview / build:** `just docs-serve` / `just docs-build`
- **Clean build artifacts:** `just clean`
- **Use a specific Python:** pass nox's `--force-python`, e.g. `just install --force-python 3.12`
- **Run one nox session:** `uv run nox -s <session>`, list them with `uv run nox --list`

## PR & Commit Conventions

Expand All @@ -93,7 +95,7 @@ lives under `pyiceberg/`, organized by concern rather than by engine:
- Keep the first version of a PR minimal; defer optimizations and edge cases to follow-ups.
- Commit messages explain the *what* and *why*, not line-by-line implementation. Be as succinct as possible.
- The Apache License header is required on every new source file (enforced by `./dev/check-license`).
- Run `make lint` and `make test` before pushing; CI runs both plus the lockfile check.
- Run `just check` and `just test` before pushing; CI runs both plus the lockfile check.

## GitHub Write Approval

Expand Down
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ exclude pyiceberg/avro/decoder_fast.c
recursive-include tests *

# Include development files
include Makefile
include justfile
include noxfile.py
include uv.lock
recursive-include dev *

Expand Down
Loading
Loading