Replace static egress resource registry with master data - #66
Open
bencehezso wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The egress registry was two hardcoded dicts - one in
utils_egress_aws.py, one inutils_egress_azure.py- holding which resource types get assessed, their category, their report label, and the strategy that sizes them. Adding a data-bearing service or retuning an existing one meant a code change and a release, even though the catalogue listing those same services already ships in master data. The registry now comes fromresourcetype_datajoined toresourcetype, with aparamsJSON column carrying the call and sizing spec.{ "enumeration": { "service": "ec2", "operation": "describe_volumes", "result_path": ["Volumes"] }, "sizing": { "id_field": "VolumeId", "name_tag": "Name", "size_field": "Size", "size_unit": "GiB", "flags": ["allocated (upper bound)"] } }What changes
load_egress_registry(csp)joins the two tables and keys byresourcetype.code, lowercased for Azure because it matches against the ARMresource.type, verbatim for AWS because collectors use it as the row identifier.paramsis merged first so the canonicalcategory,labelandstrategycannot be shadowed by a stray key.resourcetype.statusis deliberately not filtered: EBS Snapshots and Managed Disks ship as'f'to stay out of the inventory catalogue, but they still hold data that has to be egressed, and filtering on it would silently drop a whole category from the estimate.OwnerIds: ["self"]filter on EBS snapshots is now data too — without itdescribe_snapshotsreturns every publicly shared snapshot in the region.list_item_sizeandnot_sizeablereplace the three that hardcoded their own field names, takingid_field,size_field,size_unit, flags and notes from params. EFS, FSx and S3 Glacier were added afterwards with no code at all. The old strategy names are kept as aliases in_STRATEGY_COLLECTORS, so the master data migration is additive and needs no coordinated deploy.egress_inventoryandegress_inventory_tierinassessment.db, following theresource_inventorypattern, with the report reading from there.is_archiveon the tier row turns the archive rollup into a SQL sum.egress_estimate.jsonbecomesegress_inventory_raw_data.jsonto match the other raw dumps, dropslabel(it isresourcetype.name) and publishestypeascode.