Skip to content

chore(deps): drop redundant transitive pins from requirements#6110

Merged
Yicong-Huang merged 5 commits into
apache:mainfrom
Yicong-Huang:chore/drop-transitive-pins
Jul 17, 2026
Merged

chore(deps): drop redundant transitive pins from requirements#6110
Yicong-Huang merged 5 commits into
apache:mainfrom
Yicong-Huang:chore/drop-transitive-pins

Conversation

@Yicong-Huang

@Yicong-Huang Yicong-Huang commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Audits the explicit transitive pins in amber/requirements.txt and removes the two that serve no purpose. Verified with the pip dependency graph (Required-by) and a full pip install --dry-run --report resolution:

Pin Verdict Why
python-dateutil==2.8.2 Drop Required by six installed packages (pandas, botocore, betterproto, pg8000, aiobotocore, strictyaml) — always installed regardless. A pip freeze leftover from the original requirements.txt (#1259); every dependent declares >=2.8, so the pin only held us on a 2021 release. Resolves to 2.9.0.post0
aiobotocore==3.7.0 Drop Hard dependency of s3fs; its version is already uniquely constrained by the botocore==1.42.90 pin (aiobotocore's per-release botocore window is very narrow). Removing it eliminates a manual pairing hazard when bumping the boto3/botocore pins. Still resolves to 3.7.0
s3fs, SQLAlchemy, pg8000 Keep Empty Required-by — pyiceberg only pulls them via extras, so dropping them would actually uninstall them (Iceberg S3 FileIO / SqlCatalog would break at runtime). pg8000 deliberately replaces LGPL psycopg2 (#3299)
setuptools==80.10.2 Keep Guards fs's pkg_resources import against newer setuptools removing it (#4199); fs declares setuptools with no version bound

amber/LICENSE-binary-python updates python-dateutil to the resolved 2.9.0.post0 (it stays installed transitively, so the check_binary_deps.py gate still requires the claim); aiobotocore already claims 3.7.0, which is unchanged by the resolution.

Any related issues, documentation, discussions?

Closes #6108. Follow-up to #6101 / #6103.

How was this PR tested?

  • pip install --dry-run --ignore-installed --report without the pins: resolution unchanged except python-dateutil → 2.9.0.post0; aiobotocore/botocore/boto3 stay at their current versions.
  • Upgraded the local venv to the resolved versions and ran the existing pyamber suite (pytest -m "not integration") — passes.
  • CI's pyamber leg re-resolves from scratch and check_binary_deps.py verifies the LICENSE-binary-python claims.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-fable-5)

🤖 Generated with Claude Code

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file pyamber labels Jul 4, 2026
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @xuang7
    You can notify them by mentioning @xuang7 in a comment.

@codecov-commenter

codecov-commenter commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.42%. Comparing base (f722c26) to head (9696754).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6110      +/-   ##
============================================
- Coverage     70.44%   70.42%   -0.03%     
  Complexity     3404     3404              
============================================
  Files          1142     1142              
  Lines         44876    44876              
  Branches       4955     4955              
============================================
- Hits          31613    31602      -11     
- Misses        11619    11632      +13     
+ Partials       1644     1642       -2     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø) Carriedforward from ee062a2
agent-service 76.76% <ø> (ø) Carriedforward from ee062a2
amber 66.66% <ø> (ø) Carriedforward from ee062a2
computing-unit-managing-service 17.72% <ø> (ø) Carriedforward from ee062a2
config-service 52.30% <ø> (ø) Carriedforward from ee062a2
file-service 66.80% <ø> (ø) Carriedforward from ee062a2
frontend 69.24% <ø> (-0.06%) ⬇️ Carriedforward from ee062a2
notebook-migration-service 78.94% <ø> (ø) Carriedforward from ee062a2
pyamber 91.53% <ø> (ø)
workflow-compiling-service 55.14% <ø> (ø) Carriedforward from ee062a2

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main f722c26 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 378 0.231 25,425/36,747/36,747 us 🔴 +13.5% / 🔴 +155.9%
🔴 bs=100 sw=10 sl=64 946 0.577 102,629/126,950/126,950 us 🔴 +8.8% / 🔴 +28.1%
bs=1000 sw=10 sl=64 1,109 0.677 892,772/965,837/965,837 us ⚪ within ±5% / ⚪ within ±5%
Baseline details

Latest main f722c26 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 378 tuples/sec 424 tuples/sec 831.47 tuples/sec -10.8% -54.5%
bs=10 sw=10 sl=64 MB/s 0.231 MB/s 0.259 MB/s 0.507 MB/s -10.8% -54.5%
bs=10 sw=10 sl=64 p50 25,425 us 22,395 us 11,844 us +13.5% +114.7%
bs=10 sw=10 sl=64 p95 36,747 us 36,536 us 14,359 us +0.6% +155.9%
bs=10 sw=10 sl=64 p99 36,747 us 36,536 us 18,888 us +0.6% +94.6%
bs=100 sw=10 sl=64 throughput 946 tuples/sec 998 tuples/sec 1,089 tuples/sec -5.2% -13.1%
bs=100 sw=10 sl=64 MB/s 0.577 MB/s 0.609 MB/s 0.665 MB/s -5.3% -13.2%
bs=100 sw=10 sl=64 p50 102,629 us 99,079 us 92,503 us +3.6% +10.9%
bs=100 sw=10 sl=64 p95 126,950 us 116,732 us 99,121 us +8.8% +28.1%
bs=100 sw=10 sl=64 p99 126,950 us 116,732 us 112,422 us +8.8% +12.9%
bs=1000 sw=10 sl=64 throughput 1,109 tuples/sec 1,108 tuples/sec 1,117 tuples/sec +0.1% -0.7%
bs=1000 sw=10 sl=64 MB/s 0.677 MB/s 0.676 MB/s 0.682 MB/s +0.1% -0.7%
bs=1000 sw=10 sl=64 p50 892,772 us 899,666 us 910,046 us -0.8% -1.9%
bs=1000 sw=10 sl=64 p95 965,837 us 954,393 us 950,716 us +1.2% +1.6%
bs=1000 sw=10 sl=64 p99 965,837 us 954,393 us 985,832 us +1.2% -2.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,528.85,200,128000,378,0.231,25424.75,36747.48,36747.48
1,100,10,64,20,2114.23,2000,1280000,946,0.577,102628.90,126950.49,126950.49
2,1000,10,64,20,18027.74,20000,12800000,1109,0.677,892772.30,965837.40,965837.40

@Yicong-Huang
Yicong-Huang requested review from aicam and mengw15 July 4, 2026 23:54
@Yicong-Huang

Copy link
Copy Markdown
Contributor Author

cc @SarahAsad23 for review

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Yicong-Huang
Yicong-Huang force-pushed the chore/drop-transitive-pins branch from f3f7acf to 89eb663 Compare July 5, 2026 17:16
@Yicong-Huang

Copy link
Copy Markdown
Contributor Author

@aicam @mengw15 and @SarahAsad23 please help on review this. thanks

Signed-off-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com>
Signed-off-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com>
@SarahAsad23

Copy link
Copy Markdown
Contributor

LGTM!

@mengw15 mengw15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one comment on the failing license gate

Comment thread amber/LICENSE-binary-python

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Yicong-Huang and others added 2 commits July 17, 2026 12:26
Signed-off-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com>
… resolved version

python-dateutil is still installed transitively (pandas, botocore), so the
license gate requires it to stay claimed; update the pin-less resolved
version to 2.9.0.post0 instead of removing the line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Yicong-Huang
Yicong-Huang enabled auto-merge July 17, 2026 21:15
@Yicong-Huang
Yicong-Huang added this pull request to the merge queue Jul 17, 2026
Merged via the queue into apache:main with commit 1366b87 Jul 17, 2026
25 checks passed
@Yicong-Huang
Yicong-Huang deleted the chore/drop-transitive-pins branch July 17, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Drop redundant transitive pins from amber requirements

5 participants