Skip to content

Add id and name tag stores, not yet used (dense store - part 5) - #13006

Open
dougqh wants to merge 12 commits into
dougqh/tagmap-hash-supportfrom
dougqh/tagmap-store-classes
Open

dougqh wants to merge 12 commits into
dougqh/tagmap-hash-supportfrom
dougqh/tagmap-store-classes

Conversation

@dougqh

@dougqh dougqh commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds the two tag stores the span will own, without using them yet.

  • IdStore keeps known tags by id. Its one implementation today is HashIdStore, built through IdStoreFactory. A JVM loads a single IdStore subclass, so class hierarchy analysis devirtualizes every call on it.
  • HashStore keeps custom tags by name.
  • Both reuse HashSupport's bucket mechanics (Extract TagMap's bucket mechanics into HashSupport (dense store - part 4) #13002) and read through to a frozen parent's store. Each hides an inherited tag with a tombstone:
    • in the id store, one shared, immutable Entry per known tag (Entry.tombstoneOf);
    • in the name store, a set of removed names.
  • HashIdStore has no slot masks. Its buckets are the exact state.
  • Their verbs follow TagStore (Introduce TagStore for eager tag post-processors (dense store - part 3) #13001): values move by load, loadString and store, entries by getEntry, put and remove. The stores are the raw layer TagStore reads and writes, so the span's TagStore methods delegate verb for verb, and a call on a store doesn't read like TagMap's Map surface (get/set) or the span's intercepted setTag.

Nothing uses the stores yet. TagMap and the span keep their current storage. The only TagMap changes are to the nested Entry, EntryReader and BucketGroup classes, which the stores share: a tombstone flag and a lookup of a bucket chain by id.

Also in this PR:

  • HashSupport.checkBuckets now indexes a group's entries by entry instead of by bucket, so the integrity check really inspects groups.
  • The tag registry gains _dd.agent_psr as a trace-level known tag.

Motivation

This is the first of three PRs that replace the span's single TagMap with two stores split by tag kind. Here the stores arrive on their own, pinned by IdStoreContractTest, so they can be reviewed against a contract before anything depends on them.

Additional Notes

  • No behavior change: TagMap is untouched apart from the nested classes above, and the existing suites pass unchanged.
  • IdStoreContractTest runs against the stores directly. Its cases that need a TagMap view arrive with the facade PR.
  • HashStoreTest covers the name store: read-through to a parent, a local tag shadowing the parent's, removing an inherited tag (which hides it, and makes a later getAndPut report no prior value), forEach skipping shadowed and removed parent tags, and putAllLocal.
  • The serializer's tag walk stays as it was. HashSupport.forEach keeps the unrolled BucketGroup walk with no tombstone check, because TagMap serializes every span through it and only the id store ever holds a tombstone. HashIdStore walks with forEachLive, which skips tombstones, and only while its hasTombstones hint is set.
  • An id store's cursor is one allocation: it is its own bucket walker.
  • Some duplication here is temporary. TagMap's own read-through code (visibleParentCount, the parent walks) goes away in Make TagMap a facade over an id store and a name store (dense store - part 6) #13007, when TagMap becomes a facade over these stores. IdStore's split between getEntryReader and getEntry exists for DenseIdStore (Add a dense known-tag store behind an experimental flag (dense store - part 8) #13004), a store with no Entry per tag.

Where this is going

This PR is one step in replacing the span's single TagMap with storage that knows each tag's kind. The goal is to cut what every span allocates on the application thread, without changing behavior.

The end state

Later, not in this stack: cheaper read-through to parent spans, trace-level tags in a per-trace holder (TraceSegment), and primitive tag values without boxing.

The stack. This replaces #12918, split into pieces that can each be reviewed, and measured, on their own.

PR Step Allocation per 7-span trace (flag off unless noted)
#12999 A null tag value removes the tag in TagMap. This is the stack's only behavior change, kept separate on purpose. unchanged
#12992 Bucket known tags by slot in TagMap, and add TraceShapeBenchmark −96 B vs master
#13001 TagStore for the eager post-processors unchanged
#13002 Extract TagMap's bucket mechanics into HashSupport (a pure move) unchanged
#13006 Add the id and name tag stores, not yet used unchanged
#13007 Make TagMap a facade over the two stores unchanged
#13003 Have the span own its tag stores directly +104 B (+1.3%)
#13004 Dense id store behind trace.experimental.dense.tags.enabled (off by default) flag on: −30.6%
#12926 Turn the dense store on by default, add sizing hints and bulk prototype seeding flag on: about 3% less again

Every PR after #12999 is behavior-preserving.

The pieces' allocation figures come from a deterministic per-thread probe (ThreadMXBean allocated bytes) that builds the same 7-span trace as TraceShapeBenchmark. The #12992 and #12926 figures come from TraceShapeBenchmark itself.

🤖 Generated with Claude Code

@datadog-datadog-us1-prod

This comment has been minimized.

@dougqh dougqh changed the title Add id and name tag stores, not yet used Add id and name tag stores, not yet used (dense store - part 5) Oct 23, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.86 s 14.74 s [-0.0%; +1.7%] (no difference)
startup:insecure-bank:tracing:Agent 13.62 s 13.71 s [-1.5%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.07 s 16.99 s [-0.3%; +1.3%] (no difference)
startup:petclinic:iast:Agent 16.19 s 17.01 s [-8.9%; -0.7%] (maybe better)
startup:petclinic:profiling:Agent 16.65 s 16.88 s [-2.3%; -0.6%] (maybe better)
startup:petclinic:sca:Agent 17.08 s 16.96 s [-0.2%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.19 s 16.14 s [-0.5%; +1.2%] (no difference)

Commit: b6608e59 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@bric3
bric3 added this pull request to stack #13019 October 23, 2026 16:02
@dougqh
dougqh force-pushed the dougqh/tagmap-store-classes branch from ec6ec40 to 30e8d9b Compare October 23, 2026 16:57
IdStore keeps known tags by id; its one implementation today is
HashIdStore, built through IdStoreFactory so a JVM loads a single
IdStore subclass and calls on it stay devirtualized. HashStore keeps
custom tags by name. Both reuse HashSupport's bucket mechanics and read
through to a frozen parent's store, hiding an inherited tag with a
tombstone: one shared, immutable Entry per known tag (Entry.tombstoneOf)
in the id store, and a removed-name set in the name store. The hash id
store's buckets are its exact state, so it needs no slot masks.

Nothing uses the stores yet: TagMap and the span keep their current
storage. IdStoreContractTest pins the stores' behavior directly. The
only TagMap changes are to its nested Entry, EntryReader and
BucketGroup, which the stores share: the tombstone flag and a lookup of
a bucket chain by id. HashSupport.checkBuckets now indexes a group's
entries by entry rather than by bucket, so the integrity check really
inspects groups. The tag registry gains _dd.agent_psr as a trace-level
known tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/tagmap-store-classes branch from 30e8d9b to 96c49d0 Compare October 26, 2026 15:04
dougqh and others added 5 commits October 26, 2026 12:58
The stores' value methods were get/set, the verbs of TagMap's Map surface
and the span's intercepted setTag, though the stores are the raw layer
SpanStorage reads and writes. Values now move by load, loadString and
store, so the span's SpanStorage methods delegate verb for verb; entries
keep getEntry, put and remove. HashStore.get, which answers an entry,
becomes getEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…asses

# Conflicts:
#	internal-api/src/main/java/datadog/trace/api/HashSupport.java
@dougqh
dougqh requested review from a team, PerfectSlayer, bric3 and mhlidd October 27, 2026 19:57
@dougqh
dougqh marked this pull request as ready for review October 27, 2026 20:02
@dougqh
dougqh requested review from a team as code owners October 27, 2026 20:02
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-27T20:02:53.009653Z fefa3ea Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

dougqh and others added 6 commits October 27, 2026 16:33
HashSupport.forEach checked every entry's tombstone flag and walked
bucket groups through the _entryAt switch, on the path the serializer
takes for every span, though only HashIdStore ever holds a tombstone.
forEach goes back to the unrolled BucketGroup walk with no tombstone
check, for TagMap and HashStore. HashIdStore uses the new forEachLive,
which skips tombstones, and only while its hasTombstones hint is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HashCursor held a separate BucketWalker, so a cursor cost two
allocations where TagMap's iterator costs one. It now extends the
walker, whose step becomes nextEntry so it can't be mistaken for the
cursor's filtering next, and next() returns Entry covariantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coverage check flagged IdStoreFactory, which no test built a store
through, and HashSupport.BucketWalker, which only a small cursor test
reached. IdStoreFactoryTest builds stores through the JVM's factory,
checking behavior rather than the implementation, and a contract test
walks a cursor over every bucket shape and a parent level. HashStoreTest
grows to the typed setters, removal, copying, merging, sizing and a
read-through over bucket groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant