Skip to content

Sync supported assets to dataset pages and refresh docs from the API - #2653

Merged
AlexCatarino merged 1 commit into
QuantConnect:masterfrom
AlexCatarino:sync-dataset-page-supported-assets
Aug 28, 2026
Merged

Sync supported assets to dataset pages and refresh docs from the API#2653
AlexCatarino merged 1 commit into
QuantConnect:masterfrom
AlexCatarino:sync-dataset-page-supported-assets

Conversation

@AlexCatarino

Copy link
Copy Markdown
Member

Summary

Two generator changes that keep the dataset pages, the docs, and the repo's generated tables from drifting apart.

sync_supported_assets.py pushes the supported-securities tables to the dataset pages. Resources/datasets/supported-securities/ is regenerated weekly from the symbol-properties database, but the same tables are the "Supported Assets" section of a dozen dataset listings, where they were pasted by hand. They had fallen behind — Binance listed 3,320 pairs against 3,651 generated ones. The script renders each section from the repo files and writes it through market/sections, keeping the Introduction sentence, the Data Summary "Asset Coverage" row and the sidebar reach in step by substituting the number rather than the sentence. It only overwrites sections it maps, never creates one, and re-reads to confirm every write. Its workflow reports on a PR and pushes on a merge to master, so the weekly generator PR shows the diff before it reaches the live site.

The alternative-dataset generators no longer take page content from the S3 dump. They were built entirely from alternative-data-dump-v2024-01-02.json, which only changes when someone re-uploads it, so listing edits never reached the docs. get_dataset_listings() splits the two jobs the dump was doing: it still says which datasets are public, because nothing in the API records that, while content comes from market/sections/read. Drift between the two sources is printed rather than acted on.

code_generators.yml gains QUANTCONNECT_ORGANIZATION_ID, which market/sections/read requires on every call.

Test plan

  • sync_supported_assets.py --dry-run against the live pages — reports 13 in sync, 1 held
  • Stale path exercised against a doctored copy of the tables: ticker delta, count rewrite in all three places, and the reach value
  • Write-and-verify path exercised for real with an identical-bytes payload — update → re-read → diff all run, page unchanged
  • get_dataset_listings() diffed against the dump: of the listings in both, every section byte-identical except the pages synced this week
  • Alternative-Datasets-Code-Generator.py — 16 files changed (the synced pages' content), no additions or deletions, no renumbering
  • Alternative-Datasets-Skill-Generator.py — no diff
  • skill-templates/bundle-skills.py — builds all 33 skills

Generated output is not included; it lands through the weekly "Update auto-generated code" PR as usual.

Notes

algoseek-us-future-options is deliberately held out of the sync: futureoption/supported-contracts.html reads "(16) Futures Options" above 15 items, because future-table-code-generator.py counts len(FUTURE_OPTIONS) while DC is dropped — 'Dairy' is missing from the category list in get_data(). The live "(15)" is correct, so nothing is pushed until the generator is fixed.

Resources/datasets/supported-securities is regenerated weekly from the
symbol-properties database, but the same tables are the "Supported Assets"
section of a dozen dataset pages, where they were pasted by hand. The pages
drifted behind the repo silently: Binance listed 3,320 pairs against 3,651
generated ones.

sync_supported_assets.py renders each page's section from the repo files and
writes it through the market/sections endpoints, keeping the Introduction,
Data Summary and sidebar coverage counts in step by substituting the number
rather than the sentence. It only overwrites the sections it maps, never
creates one, and re-reads to confirm every write. Its workflow reports on a
pull request and pushes on a merge to master, so the generator PR shows the
diff before it reaches the live website.

The alternative-dataset generators had the opposite problem: they were built
entirely from alternative-data-dump-v2024-01-02.json, a snapshot that only
changes when someone re-uploads it, so edits to a listing never reached the
docs. get_dataset_listings() splits the two things the dump was doing. The
dump still says which datasets are public, because nothing in the API records
that. Page content comes from market/sections/read, so a listing edit shows
up on the next run. Drift between the two sources is printed rather than
acted on, since either direction needs a person to decide.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AlexCatarino
AlexCatarino force-pushed the sync-dataset-page-supported-assets branch from 2d238f4 to a5c84bf Compare August 28, 2026 18:26
@AlexCatarino
AlexCatarino merged commit acf6c1c into QuantConnect:master Aug 28, 2026
1 check passed
@AlexCatarino
AlexCatarino deleted the sync-dataset-page-supported-assets branch August 28, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant