Skip to content

ci(fees): add monthly fee schedule check workflow #502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pigbibi merged 1 commit into from
Nov 1, 2026
Merged

ci(fees): add monthly fee schedule check workflow #502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 242 additions & 0 deletions .github/workflows/fee-schedule-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
name: Fee schedule check

# Monthly review of the Firstrade fee schedule (tracking issue #497).
#
# check: compares the SEC Section 31 rate and the FINRA TAF rates on the official sites with
# application/firstrade_fee_schedule.py. Any fetch/parse failure is an error, never a pass.
# bump: only when every value is unchanged. Pushes a branch that moves `verified_through` to the
# end of next month (plus the test pin and docs) and opens a PR. It never merges.
# notify: when rates changed, the check failed, or the PR could not be opened. Opens or updates one
# issue. Rates are never edited automatically.
#
# Only GITHUB_TOKEN is used. Pull requests opened with it do not start other workflows, so the
# bump job runs the relevant tests itself and a reviewer closes and reopens the PR to run CI.
# Optional repository variable FEE_CHECK_CONTACT: a monitored address that is sent to sec.gov in
# the User-Agent (SEC fair-access policy).

on:
schedule:
- cron: '17 2 22 * *'
workflow_dispatch:
pull_request:
paths:
- '.github/workflows/fee-schedule-check.yml'
- 'scripts/check_fee_schedule.py'

permissions: {}

concurrency:
group: fee-schedule-check-${{ github.event_name == 'pull_request' && github.head_ref || 'main' }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

env:
ISSUE_TITLE: 'Firstrade fee schedule check needs attention'

jobs:
check:
name: Check official sources
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
outputs:
status: ${{ steps.run.outputs.status }}
report: ${{ steps.run.outputs.report }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
persist-credentials: false

- name: Ensure pdftotext
run: |
set -euo pipefail
command -v pdftotext >/dev/null || {
sudo apt-get update -qq
sudo apt-get install -y -qq --no-install-recommends poppler-utils
}

- name: Compare with sec.gov and finra.org
id: run
env:
FEE_CHECK_CONTACT: ${{ vars.FEE_CHECK_CONTACT }}
run: |
set -uo pipefail
rc=0
python3 -m scripts.check_fee_schedule check \
--report "$RUNNER_TEMP/report.md" --json "$RUNNER_TEMP/result.json" || rc=$?
status=error
case "$rc" in
0) status=unchanged ;;
2) status=changed ;;
esac
[ -s "$RUNNER_TEMP/report.md" ] || printf 'The check produced no report.\n' > "$RUNNER_TEMP/report.md"
cat "$RUNNER_TEMP/report.md" >> "$GITHUB_STEP_SUMMARY"
delimiter="EOF_$(openssl rand -hex 8)"
{
echo "status=$status"
echo "report<<$delimiter"
cat "$RUNNER_TEMP/report.md"
echo "$delimiter"
} >> "$GITHUB_OUTPUT"

- name: Fail pull request runs unless the sources match
if: github.event_name == 'pull_request' && steps.run.outputs.status != 'unchanged'
run: |
echo "The check status is ${{ steps.run.outputs.status }}; see the job summary." >&2
exit 1

bump:
name: Open verified_through bump
needs: check
if: >-
github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
&& needs.check.outputs.status == 'unchanged'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: write
pull-requests: write
outputs:
result: ${{ steps.bump.outputs.result }}
branch: ${{ steps.bump.outputs.branch }}
pr_url: ${{ steps.bump.outputs.pr_url }}
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
REPORT: ${{ needs.check.outputs.report }}
# The bump rewrites a module that it imported; a stale .pyc would hide the change from pytest.
PYTHONDONTWRITEBYTECODE: '1'
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
persist-credentials: false

- name: Bump verified_through and open a pull request
id: bump
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
gh auth setup-git

python3 -m scripts.check_fee_schedule bump --output "$RUNNER_TEMP/new_date"
new_date="$(cat "$RUNNER_TEMP/new_date")"
if [ -z "$new_date" ]; then
echo "result=noop" >> "$GITHUB_OUTPUT"
exit 0
fi
branch="chore/fee-schedule-verified-through-$new_date"
echo "branch=$branch" >> "$GITHUB_OUTPUT"

# The bump rewrites only the date pins; run the tests that cover them.
python3 -m venv "$RUNNER_TEMP/venv"
"$RUNNER_TEMP/venv/bin/pip" install --quiet pytest==9.1.1
"$RUNNER_TEMP/venv/bin/python" -m pytest -q \
tests/test_statement_reconciliation.py tests/test_check_fee_schedule.py
git diff --stat

# Never overwrite an existing branch.
if git ls-remote --exit-code --heads origin "$branch" >/dev/null; then
if [ -n "$(gh pr list --head "$branch" --state open --json number --jq '.[].number')" ]; then
echo "result=exists" >> "$GITHUB_OUTPUT"
else
echo "result=branch-only" >> "$GITHUB_OUTPUT"
fi
exit 0
fi

git switch -c "$branch"
git add -u
git commit -q -m "chore(fees): extend fee schedule verified_through to $new_date" \
-m "Scheduled fee-schedule-check found the SEC Section 31 and FINRA TAF rates unchanged." \
-m "No rate row was changed. Refs #497."
git push origin "$branch"

{
echo "## AI assistance"
echo
echo "None. Opened by the scheduled \`Fee schedule check\` workflow, which runs"
echo "\`scripts/check_fee_schedule.py\`. A human must review before merge."
echo
echo "## What changed"
echo
echo "Moves \`verified_through\` in \`application/firstrade_fee_schedule.py\` to \`$new_date\`"
echo "so sells dated up to then keep reconciling instead of returning \`unsupported\`. No"
echo "rate row was changed. The test review pin and the doc mentions move with it."
echo
echo "Refs #497 (recurring review; intentionally not closed)."
echo
echo "## How verified"
echo
echo "Workflow run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID."
echo "\`pytest tests/test_statement_reconciliation.py tests/test_check_fee_schedule.py\`"
echo "passed in that run. CI did not start on this pull request because it was opened"
echo "with \`GITHUB_TOKEN\`: close and reopen it to run the full checks."
echo
printf '%s\n' "$REPORT" | tail -n +2 | sed 's/^## /### /'
} > "$RUNNER_TEMP/pr_body.md"

if pr_url="$(gh pr create --base main --head "$branch" \
--title "chore(fees): extend fee schedule verified_through to $new_date" \
--body-file "$RUNNER_TEMP/pr_body.md")"; then
echo "result=pr-opened" >> "$GITHUB_OUTPUT"
echo "pr_url=$pr_url" >> "$GITHUB_OUTPUT"
else
echo "::warning::The branch was pushed but the pull request could not be created."
echo "result=branch-only" >> "$GITHUB_OUTPUT"
fi

notify:
name: Open or update the attention issue
needs: [check, bump]
if: >-
always() && github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
&& (needs.check.result != 'success' || needs.check.outputs.status != 'unchanged'
|| needs.bump.result == 'failure' || needs.bump.outputs.result == 'branch-only')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
issues: write
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
CHECK_RESULT: ${{ needs.check.result }}
CHECK_STATUS: ${{ needs.check.outputs.status }}
BUMP_RESULT: ${{ needs.bump.result }}
BUMP_OUTCOME: ${{ needs.bump.outputs.result }}
BUMP_BRANCH: ${{ needs.bump.outputs.branch }}
REPORT: ${{ needs.check.outputs.report }}
steps:
- name: Open or update the issue
run: |
set -euo pipefail
run_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
{
echo "Fee schedule check, workflow run: $run_url"
echo
if [ "$CHECK_RESULT" != "success" ]; then
echo "The check job did not finish (result: $CHECK_RESULT). Nothing was changed."
elif [ "$CHECK_STATUS" != "unchanged" ]; then
echo "Status: **$CHECK_STATUS**. Nothing was changed and no bump pull request was opened."
else
echo "The rates are unchanged, but the \`verified_through\` bump could not be completed"
echo "(bump job: $BUMP_RESULT, outcome: ${BUMP_OUTCOME:-none})."
if [ -n "$BUMP_BRANCH" ]; then
echo
echo "Branch \`$BUMP_BRANCH\` is on the remote. Open a pull request from it:"
echo "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/compare/main...$BUMP_BRANCH?expand=1"
fi
fi
echo
printf '%s\n' "$REPORT" | tail -n +2 | sed 's/^## /### /'
echo
echo "Tracking issue: #497. Rates are never changed automatically."
} > "$RUNNER_TEMP/issue_body.md"

number="$(gh issue list --state open --limit 100 --json number,title \
| jq -r --arg title "$ISSUE_TITLE" '[.[] | select(.title == $title)][0].number // empty')"
if [ -n "$number" ]; then
gh issue comment "$number" --body-file "$RUNNER_TEMP/issue_body.md"
else
gh issue create --title "$ISSUE_TITLE" --body-file "$RUNNER_TEMP/issue_body.md"
fi
25 changes: 25 additions & 0 deletions docs/statement-reconciliation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,31 @@ until 60 days after the FY2027 appropriation is enacted) and the FINRA filings
above are immediately effective but still open to SEC suspension. Re-verify the
sources and move `verified_through` forward in the same change.

The `Fee schedule check` workflow (`.github/workflows/fee-schedule-check.yml`) does
this review every month, on the 22nd at 02:17 UTC, and on demand through
`workflow_dispatch`. `scripts/check_fee_schedule.py check` reads the SEC Section 31
advisories and SEC filing pages, and the FINRA fee adjustment schedule, rule filing
listing and filing PDFs, then compares them with the schedule. A fetch or parse
failure is an error, never a pass.

- Everything matches: the `bump` job moves `verified_through` to the end of the next
month (plus the "checked" date in the module docstring, the test review pin
`VERIFIED_THROUGH` and the two doc mentions), runs the tests and opens a pull
request. It never merges.
- A rate changed, a new FINRA fee filing appeared, an SEC page mentions a
suspension, or the check failed: the `notify` job opens or updates one issue
titled "Firstrade fee schedule check needs attention". Rates are never edited
automatically; update the segments in a normal pull request.

The workflow uses `GITHUB_TOKEN` only. Pull requests it opens do not start CI, so
close and reopen the pull request to run the checks before merging. If the
repository setting that lets Actions create pull requests is off, the job leaves
the branch on the remote and the issue carries a compare link instead. Set the
repository variable `FEE_CHECK_CONTACT` to a monitored address: sec.gov rejects
clients that do not identify themselves. The script is tied to the current page
layouts and to the filings named in `REVIEWED_FINRA_FILINGS`; when a layout
changes the check fails loudly and the parser needs an update.

## Valuation basis difference

The balances endpoint and the positions endpoint price the same shares from
Expand Down
2 changes: 2 additions & 0 deletions docs/statement-reconciliation.zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@

`verified_through`（当前 `2026-12-01`）是费率最后一次对照公开来源核实的交易日。之后的交易日为 `unsupported`（`..._not_verified_after_<日期>`）：SEC 2027 财年 Section 31 费率尚未公布（FY2026 费率在 2027 财年拨款法案生效后 60 天内继续有效），上述 FINRA 备案虽即时生效但仍可能被 SEC 暂停。更新来源核实时需在同一变更中推进 `verified_through`。

`Fee schedule check` 工作流（`.github/workflows/fee-schedule-check.yml`）每月 22 日 UTC 02:17 自动执行这项复核，也可手动触发（`workflow_dispatch`）：`scripts/check_fee_schedule.py check` 读取 SEC Section 31 公告和 FINRA 费率表、备案列表与备案 PDF，与费率表比对；抓取或解析失败一律视为错误，不会当作通过。全部一致时 `bump` 任务把 `verified_through` 推到下月末（并同步测试里的复核日期和文档），跑完测试后开 PR，不会自动合并。费率有变化、出现新的 FINRA 费用备案、SEC 页面提到暂停或检查失败时，`notify` 任务开或更新一个 issue，费率不会被自动修改。仅使用 `GITHUB_TOKEN`，它开的 PR 不会触发 CI，合并前请关闭再重新打开 PR 以运行检查。建议把仓库变量 `FEE_CHECK_CONTACT` 设为可联系的邮箱，sec.gov 会拒绝不表明身份的请求。

## 估值口径差异

余额接口与持仓接口对同一批股票使用不同报价口径，因此在数量和现金一致时，`long_stock_value`（余额接口）与持仓 `market_value` 之和可能相差几分钱。`valuation_basis_difference` 把它单独报告：
Expand Down
Loading
Loading