Skip to content

Add policy-gated ETW logging for sidecar GCS - #2982

Open
Manish Ranjan Mahanta (marma-dev) wants to merge 5 commits into
microsoft:mainfrom
marma-dev:sidecar-gcs-etw-logging
Open

Manish Ranjan Mahanta (marma-dev) wants to merge 5 commits into
microsoft:mainfrom
marma-dev:sidecar-gcs-etw-logging

Conversation

@marma-dev

@marma-dev Manish Ranjan Mahanta (marma-dev) commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Emit sidecar GCS diagnostics through a dedicated Microsoft.Virtualization.GCSSidecar ETW provider using the existing etwlogrus hook.
  • Keep emission disabled until the validated confidential policy permits allow_runtime_logging; reuse existing Log Forward Service provider policy for collection without adding a new permission.
  • Include the provider in default collection at Information level and merge provider-name/GUID case variants without duplicate subscriptions.

Validation performed

  • Changed-package Windows suites, targeted Rego logging-policy tests, Go vet, sidecar/host builds, and Linux shared-policy test cross-compilation passed.
  • Live VBS C-WCOW with Secure Boot enabled confirmed sidecar forwarding and continued host parsing with the timestamp fix in Handle formatting markers in forwarded ETW timestamps #2981. Runtime logging denied: zero sidecar events while other providers continued forwarding.
  • Fixed case-insensitive overrides and added canonical/lowercase/uppercase name and GUID regression coverage; provider and sidecar suites passed, and CI-pinned golangci-lint reported zero issues.
  • Ordinary non-confidential WCOW (NanoServer LTSC 2025, guest 10.0.26100.33674) booted with default options, confirmed SIDECAR_ABSENT, and completed its guest command with exit code 0.
  • Separately tested active Log Forward Service collection on the stock 10.0.28000.2178 VBS guest with the new provider absent and its canonical GUID requested: 216 events from other providers continued forwarding without a UVM startup failure.

Compatibility and coverage notes

  • Handle formatting markers in forwarded ETW timestamps #2981 fixes the host timestamp parsing issue exposed by live testing and is required for correct consumption of the sidecar events.
  • Existing policies using default sources must allow the new provider or enable provider dropping; otherwise existing fail-close behavior may reject UVM startup.
  • Ordinary-WCOW startup and active absent-provider collection were validated separately: the LTSC 2025 guest does not advertise Log Forward Service support. An ordinary forwarding-capable WCOW image has not been tested end-to-end.
  • The existing launcher output-processing teardown hang and forced-shutdown partial-record diagnostic remain outside this PR; all test UVMs are stopped.

Startup event buffering

The sidecar now retains at most 128 pre-policy events / 256 KiB (16 KiB per event). Buffered events flush only after validated allow_runtime_logging approval and ETW collector enablement; policy denial discards them. Oversized or unsupported startup records are dropped. No post-policy events are queued while collection is disabled.

Final VBS C-WCOW test (Secure Boot enabled, private Rego sidecar): 14 pre-policy sidecar startup events were delivered after Log Forward Service enabled; host parsed all without timestamp errors. With runtime logging denied: zero sidecar events, while other ETW providers continued forwarding. Unit coverage includes both policy/collector orderings, denial, event and byte bounds, oversized fields, and concurrent state changes.

Manish Ranjan Mahanta added 2 commits November 5, 2026 21:52
Signed-off-by: Manish Ranjan Mahanta <mmahanta@microsoft.com>
Signed-off-by: Manish Ranjan Mahanta <mmahanta@microsoft.com>
Comment thread cmd/gcs-sidecar/main.go Resolved
Signed-off-by: Manish Ranjan Mahanta <mmahanta@microsoft.com>
@marma-dev

Copy link
Copy Markdown
Contributor Author

Investigated the windows-2022 failure: GitHub reports "The self-hosted runner lost communication with the server." Rego policy, repo, LCOW/WCOW-tagged, non-functional, and parity test steps passed before the runner was lost during the shim-test step. The full job log blob is unavailable, so the underlying runner-disconnect cause could not be established. Pushed provider cleanup in 9f13693; a new CI run should validate it.

@apurv15

apurv15 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Shouldn't early boot/init events of gcs side car be captured but relayed only when enabled?

We could maintain a small, bounded in-memory buffer in the sidecar’s logging hook: record only startup diagnostics; flush them to ETW once the policy is validated and allows runtime logging; discard them if policy denies logging.

Signed-off-by: Manish Ranjan Mahanta <mmahanta@microsoft.com>
@marma-dev

Manish Ranjan Mahanta (marma-dev) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

apurv15 Implemented the startup-buffer suggestion in e6a6291. The sidecar buffers at most 128 pre-policy events / 256 KiB (16 KiB per event), and flushes only after both runtime-policy approval and ETW collector enablement. Policy denial discards pending events. Live VBS C-WCOW testing forwarded 14 pre-policy startup events; the denied-policy run forwarded zero sidecar events while other ETW providers continued.

}

func copyStartupEntry(entry *logrus.Entry) (*logrus.Entry, int, bool) {
if len(entry.Message) > maxStartupEventBytes {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI feedback:
Every event passes through  copyStartupEntry , even after policy approval and collector enablement. Any unsupported field drops the entire event. For example, the existing “unpacked COSE1 cert chain” diagnostic contains a  cwtClaims  map and will never reach ETW. The underlying ETW hook supports formatting such values. Restrict startup filtering to startup buffering, or preserve these fields safely during live forwarding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 580166b. Once policy is approved and ETW collection is enabled, live events now pass directly to the ETW hook; bounded copying is used only for startup-buffered entries. Added regression coverage for live cwtClaims passthrough and safe copying of buffered structured claims. Targeted Windows tests, vet, Rego build, and pinned lint passed.

Comment thread cmd/gcs-sidecar/main.go Outdated
logrus.WithError(err).Error("failed to close sidecar ETW provider")
}
}()
runtimeLogHook.Store(runtimeHook)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI feedback:
After publishing the hook, initialization can load  false , then an ETW callback sets the hook to  true , and initialization subsequently applies its stale  false . With no further notification, forwarding remains disabled despite an active collector.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 580166b. RuntimeLoggingController serializes collector state updates with hook attachment, preserving the latest enable/disable state regardless of callback ordering. Added coverage for both enable-before-attach and attach-before-enable. Targeted Windows tests, vet, Rego build, and pinned lint passed.

Signed-off-by: Manish Ranjan Mahanta <mmahanta@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants