Sync supported assets to dataset pages and refresh docs from the API - #2653
Merged
AlexCatarino merged 1 commit intoAug 28, 2026
Merged
Conversation
Resources/datasets/supported-securities is regenerated weekly from the symbol-properties database, but the same tables are the "Supported Assets" section of a dozen dataset pages, where they were pasted by hand. The pages drifted behind the repo silently: Binance listed 3,320 pairs against 3,651 generated ones. sync_supported_assets.py renders each page's section from the repo files and writes it through the market/sections endpoints, keeping the Introduction, Data Summary and sidebar coverage counts in step by substituting the number rather than the sentence. It only overwrites the sections it maps, never creates one, and re-reads to confirm every write. Its workflow reports on a pull request and pushes on a merge to master, so the generator PR shows the diff before it reaches the live website. The alternative-dataset generators had the opposite problem: they were built entirely from alternative-data-dump-v2024-01-02.json, a snapshot that only changes when someone re-uploads it, so edits to a listing never reached the docs. get_dataset_listings() splits the two things the dump was doing. The dump still says which datasets are public, because nothing in the API records that. Page content comes from market/sections/read, so a listing edit shows up on the next run. Drift between the two sources is printed rather than acted on, since either direction needs a person to decide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AlexCatarino
force-pushed
the
sync-dataset-page-supported-assets
branch
from
August 28, 2026 18:26
2d238f4 to
a5c84bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two generator changes that keep the dataset pages, the docs, and the repo's generated tables from drifting apart.
sync_supported_assets.pypushes the supported-securities tables to the dataset pages.Resources/datasets/supported-securities/is regenerated weekly from the symbol-properties database, but the same tables are the "Supported Assets" section of a dozen dataset listings, where they were pasted by hand. They had fallen behind — Binance listed 3,320 pairs against 3,651 generated ones. The script renders each section from the repo files and writes it throughmarket/sections, keeping the Introduction sentence, the Data Summary "Asset Coverage" row and the sidebarreachin step by substituting the number rather than the sentence. It only overwrites sections it maps, never creates one, and re-reads to confirm every write. Its workflow reports on a PR and pushes on a merge to master, so the weekly generator PR shows the diff before it reaches the live site.The alternative-dataset generators no longer take page content from the S3 dump. They were built entirely from
alternative-data-dump-v2024-01-02.json, which only changes when someone re-uploads it, so listing edits never reached the docs.get_dataset_listings()splits the two jobs the dump was doing: it still says which datasets are public, because nothing in the API records that, while content comes frommarket/sections/read. Drift between the two sources is printed rather than acted on.code_generators.ymlgainsQUANTCONNECT_ORGANIZATION_ID, whichmarket/sections/readrequires on every call.Test plan
sync_supported_assets.py --dry-runagainst the live pages — reports 13 in sync, 1 heldreachvalueupdate→ re-read → diff all run, page unchangedget_dataset_listings()diffed against the dump: of the listings in both, every section byte-identical except the pages synced this weekAlternative-Datasets-Code-Generator.py— 16 files changed (the synced pages' content), no additions or deletions, no renumberingAlternative-Datasets-Skill-Generator.py— no diffskill-templates/bundle-skills.py— builds all 33 skillsGenerated output is not included; it lands through the weekly "Update auto-generated code" PR as usual.
Notes
algoseek-us-future-optionsis deliberately held out of the sync:futureoption/supported-contracts.htmlreads "(16) Futures Options" above 15 items, becausefuture-table-code-generator.pycountslen(FUTURE_OPTIONS)whileDCis dropped —'Dairy'is missing from the category list inget_data(). The live "(15)" is correct, so nothing is pushed until the generator is fixed.