Skip to content

Replace static egress resource registry with master data - #66

Open
bencehezso wants to merge 2 commits into
mainfrom
task/dynamic_egress_registry
Open

Replace static egress resource registry with master data#66
bencehezso wants to merge 2 commits into
mainfrom
task/dynamic_egress_registry

Conversation

@bencehezso

Copy link
Copy Markdown
Contributor

Summary

The egress registry was two hardcoded dicts - one in utils_egress_aws.py, one in utils_egress_azure.py - holding which resource types get assessed, their category, their report label, and the strategy that sizes them. Adding a data-bearing service or retuning an existing one meant a code change and a release, even though the catalogue listing those same services already ships in master data. The registry now comes from resourcetype_data joined to resourcetype, with a params JSON column carrying the call and sizing spec.

{
  "enumeration": {
    "service": "ec2",
    "operation": "describe_volumes",
    "result_path": ["Volumes"]
  },
  "sizing": {
    "id_field": "VolumeId",
    "name_tag": "Name",
    "size_field": "Size",
    "size_unit": "GiB",
    "flags": ["allocated (upper bound)"]
  }
}

What changes

  • Registry constructionload_egress_registry(csp) joins the two tables and keys by resourcetype.code, lowercased for Azure because it matches against the ARM resource.type, verbatim for AWS because collectors use it as the row identifier. params is merged first so the canonical category, label and strategy cannot be shadowed by a stray key. resourcetype.status is deliberately not filtered: EBS Snapshots and Managed Disks ship as 'f' to stay out of the inventory catalogue, but they still hold data that has to be egressed, and filtering on it would silently drop a whole category from the estimate.
  • Enumeration comes from params — service, operation, result path and kwargs, so the row can override what its own code says. AWS Backup is why: the catalogue code enumerates backup plans for the inventory scan, while sizing needs the vaults. The OwnerIds: ["self"] filter on EBS snapshots is now data too — without it describe_snapshots returns every publicly shared snapshot in the region.
  • Two parameterised collectorslist_item_size and not_sizeable replace the three that hardcoded their own field names, taking id_field, size_field, size_unit, flags and notes from params. EFS, FSx and S3 Glacier were added afterwards with no code at all. The old strategy names are kept as aliases in _STRATEGY_COLLECTORS, so the master data migration is additive and needs no coordinated deploy.
  • The estimate is stored, not just serialisedegress_inventory and egress_inventory_tier in assessment.db, following the resource_inventory pattern, with the report reading from there. is_archive on the tier row turns the archive rollup into a SQL sum. egress_estimate.json becomes egress_inventory_raw_data.json to match the other raw dumps, drops label (it is resourcetype.name) and publishes type as code.

@bencehezso bencehezso self-assigned this Aug 14, 2026
@bencehezso bencehezso added the enhancement New feature or request label Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant