Skip to content

Independently moderate newly opened issues - #2030

Merged
Stella Huang (StellaHuang95) merged 7 commits into
microsoft:mainfrom
StellaHuang95:issue-moderation-advisor
Nov 18, 2026
Merged

Stella Huang (StellaHuang95) merged 7 commits into
microsoft:mainfrom
StellaHuang95:issue-moderation-advisor

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a single independent GitHub Agentic Workflow for newly opened issues. It starts on issues.opened, judges the report itself, and does not wait for central ~spam / ~info-needed flags. There is no hourly catch-up, flag-event router, automatic backlog sweep, or automatic AI review on edits, comments, or reopening.

Independent judgment Action
Clear gibberish, explicit tracker-only test submissions, or plainly unrelated material Fixed polite reply, then close as not_planned
Plausible extension report without enough information, including an empty extension template Friendly template request, add only local info-needed, leave open
Useful report or uncertainty No comment, label, reaction, or state change

The information request is:

Thanks for reaching out! Could you fill out the issue template in this issue? A little more information would help us investigate and better understand how to help.

Language, spelling, brevity, a new account, missing logs, or unfilled optional fields are not sufficient grounds for closure. This version creates no moderation-review-needed queue label.

Follow-up responses: comments and description edits

The existing reporter-comment cleanup now works for issues even without triage-needed; existing shared-action authorization remains, and pull requests keep their previous guard.

A small deterministic issues.edited job also treats a genuine nonempty description update by the authenticated human issue author as a response. It removes only plain info-needed. It does not run AI, post another comment, close/reopen an issue, or remove central flags. This prevents a reporter who fills out the requested template in the description from later being closed as unanswered.

  • Title-only, whitespace-only, and empty-body changes do not clear a pending request.
  • Current body/state/author/label and timeline checks reject stale responses, unverifiable ordering, renewed requests, or newer human handling; a final reread guards intervening changes. Simultaneous events with unverifiable ordering defer conservatively.
  • Actual GitHub testing showed that one title-and-body update can produce separate edited webhooks. A reporter rename matching that response's actor, timestamp, and new title does not veto its body update, even when the body webhook omits changes.title. Later or mismatched handling remains protected. Passive mentions/subscriptions are not treated as renewed requests.
  • The existing seven-day closer is unchanged. Clearing the label excludes the answered issue from its information-needed path; unanswered issues retain the existing follow-up behavior.

Independence and safeguards

  • The model receives a prepared JSON file containing title and description only. Labels, assignments, account metadata, and other bots' judgments are not classification input. Both central flags, either flag, or neither flag can be present without authorizing or vetoing a judgment.
  • Central automation remains untouched and may apply labels later. The moderator never adds/removes central flags, rewrites assignments, changes area/type classifications, or locks issues. Both systems may consume inference while they coexist.
  • A native preflight prepares content plus a trusted fingerprint before inference. Human discussion, manual triage/assignment, pins/project changes and other protected decisions, closed/locked issues, and previous workflow replies stop automatic action. Opening-payload template labels are neutral; later human label changes are not. Bot metadata churn does not create a false human override or invalidate content.
  • Publication rereads state before commenting and before labeling/closing. Changed content or intervening human handling backs off. A previous response prevents duplicate replies and reclosure after human reopening. Manual runs without an original opening payload conservatively defer human-origin labels, including possible template labels.
  • The Copilot agent has GitHub/shell/editing tools disabled, plus explicit CLI --deny-tool shell --deny-tool write arguments. It reads the whole prepared file with native view and forceReadLargeFiles: true; unavailable/incomplete input must produce a safe no-op. It emits only a classification. Fixed code chooses the issue, reply, labels and state change.
  • Strict threat detection must pass before the separate write-enabled publisher runs. Publication independently downloads the original preflight fingerprint; the agent's copy is not trusted. API errors propagate. Partial failure after a reply requires maintainer inspection rather than blind duplicate publication. GitHub API reads/writes are not atomic, so a small read/write race remains.

Operation, budgets, and rollout

Manual dispatch accepts one issue number with the same human-protection guards. It is live publication, not a dry run, not a force override, and not a backlog sweep. The committed workflow exposes no staged dispatch input.

Agent execution is serialized with a 20-turn, 10-minute, 500-credit per-run limit and a separate 200-credit detector budget. The 10,000-credit daily guardrail applies to automatic runs; the pinned runtime exempts manual dispatch. Daily accounting can lag in-flight work. Missed events or failures require manual retry/review.

The source and generated executable use gh-aw v1.2.2 and pinned runtime 53d0749e19424a918acd82b4d140264ce48f2f14. Focused safeguard tests run in existing Linux/Windows unit CI jobs, and the large lockfile is marked generated. Repository/organization policy and billing must permit copilot-requests: write and the pinned Actions. Administrative upstream Actions policy could not be inspected with the available access (HTTP 403); no settings were changed.

Review and validation

Repeated manual lifecycle review and independent code review found and fixed the description-only response gap and the same-response rename edge case. The latter was tested against actual split GitHub webhooks, not just assumed payloads. The final narrow independent review found no remaining meaningful defect in the corrections.

  • 30 focused scenario tests passed, including edit-only responses, actor/content validation, renewed requests, combined/split title-body events, passive timeline events, API failures, and existing moderation guards. In-memory mutation checks verified that the new regressions reject the corresponding broken behaviors.
  • Lint, TypeScript test compilation, and the existing unit suite passed: 3,573 passing, 9 pending.
  • Pinned workflow compilation and policy lint passed with zero compiler warnings; final no-emit compilation passed.
  • Final production helper/tests, reporter-cleanup workflow, Markdown source, and executable were compared with private candidate 770d176407659ed9c39f80ef5f893a7251cafe87. No identity substitutions were used.

Live end-to-end checks

These runs are in the private isolated fixture repository and require access. Actual content, comments, labels and states were asserted, not just green job conclusions.

Scenario Verified result / evidence
New incomplete report Friendly request and local info-needed, initially open. Run
Author fills the requested template and clarifies the title in one update Actual split title/body webhooks clear only local info-needed, leave open and add no reply. Run
Author supplies details by editing only the description Same cleanup, preserving existing ~spam / ~info-needed. Run
Title-only / whitespace-only / empty-body edits Pending info-needed retained. Title, whitespace, empty
Existing author-comment path without triage-needed Label removed; issue stayed open; original authorization logic retained. Run
Answered versus unanswered information requests Existing closer preserved the answered fixture and closed the unanswered fixture. Only the private test copy used zero days; production remains seven days. Run
64,892-character log-heavy report with the actual problem buried in the middle Final candidate returned the entire report in one native-view read; returned JSON was compared with the full prepared input. No shell fallback was attempted. Classified relevant; no comment, label, or closure. Final run
Clear junk Fixed polite reply and closure of only its own issue. Run
Human reopens that handled junk issue, then explicitly requests manual review No automatic AI review on reopening. Manual run skipped inference/publication, left it open, and posted no duplicate response. Run

The earlier independent-flow checks also verified useful French reports despite misleading Bot metadata, pre-inference human-pin protection, stale-publication rejection after a body edit, and cross-issue instruction confinement. Logs and pre-cleanup outcome snapshots are retained locally. No upstream issue was created or modified.

Keep this PR draft for maintainer inspection before making it ready for review.

@StellaHuang95 Stella Huang (StellaHuang95) added github_actions Pull requests that update GitHub Actions code no-changelog labels Nov 17, 2026
@StellaHuang95 Stella Huang (StellaHuang95) changed the title Add guarded advisory issue moderation with gh-aw Add guarded agentic issue moderation Nov 17, 2026
@StellaHuang95 Stella Huang (StellaHuang95) changed the title Add guarded agentic issue moderation Follow up on central bot issue flags Nov 17, 2026
@StellaHuang95 Stella Huang (StellaHuang95) changed the title Follow up on central bot issue flags Independently moderate newly opened issues Nov 18, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review report content independently of bot metadata, remove scheduled flag-follow-up routing, and preserve human decisions with guarded fixed replies and trusted snapshots. Verify the narrowed model runtime and independent outcomes with focused and live tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Acknowledge real author edits without another AI review, preserve renewed requests and existing comment/PR behavior, document manual budget exemptions, and request complete reads for log-heavy reports.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95
Stella Huang (StellaHuang95) marked this pull request as ready for review November 18, 2026 21:52
@rchiodo

Copy link
Copy Markdown
Contributor

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Copy link
Copy Markdown
Contributor

Reviewed commit: 08118f09e462e7c5229dc1c86da7504812e61192

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: C:\Users\rchiodo\AppData\Local\Programs\Podman\podman.EXE --connection pyrx-automation failed: stderr: Error: unable to start container "a201721345a92451af5d69422a473e7fa79858a0a2769bab0e9aa2720a4461ba": crun: open `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

Test runs: none recorded.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 18, 2026

@bschnurr Bill Schnurr (bschnurr) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

Stella Huang (StellaHuang95) added a commit that referenced this pull request Nov 19, 2026
## Summary

Follow up on #2030 to fix two
production-reproduced cases where cosmetic issue edits incorrectly
acknowledged an unanswered information request.

- Verify description responses against GitHub's actual body-edit
history, including before/after content, editor identity, and request
ordering. Title-only webhooks can contain `changes.body.from: ""`; that
field alone is not proof of a response.
- Ignore prose blank-line and line-ending changes without discarding
meaningful indentation or blank lines in Python code, including
fenced/list/indented code with tabs.
- Retry transient HTTP 500/502/503/504 failures at most twice, with
1s/2s backoff and the complete live safeguards repeated before each
attempt. Non-transient and exhausted failures still fail visibly.

The change is limited to deterministic description-response cleanup. It
does not change model classifications, add inference triggers, alter
central labels, or change the seven-day closer.

## Production evidence

Both cleanup defects were reproduced in
[#2035](#2035),
a now-closed synthetic validation fixture:

| Scenario | Existing behavior |
|---|---|
| [Title-only
update](https://github.com/microsoft/vscode-python-environments/actions/runs/44436255687)
| Removed `info-needed` despite a byte-for-byte identical description. |
| [Interior blank-line-only
update](https://github.com/microsoft/vscode-python-environments/actions/runs/44437297451)
| Removed the label without any added information. |
| [Genuine description response with an API
error](https://github.com/microsoft/vscode-python-environments/actions/runs/44436346435)
| A GitHub HTTP 500 left the label in place until the failed job was
manually rerun. |

Both false acknowledgements can exclude unanswered issues from the
information-needed closer. Conversely, an unrecovered API failure can
leave an answered issue on that path.

## Safeguards and walkthrough

- Body-edit timestamps can precede REST/webhook `updated_at` by a
second, so the fix binds actual revision snapshots rather than requiring
exact timestamp equality.
- Undeleted empty-body revisions have `diff: null`, `deletedAt: null`;
these are handled separately from missing or deleted history.
- Newer requests and human activity are compared against the actual body
response time, not a later rename/metadata timestamp.
- Current issue/body/state/label checks and the final reread remain in
place. Each retry repeats the reads, including renewed-request checks. A
DELETE that succeeded despite an error is not blindly repeated.
- Missing/unverifiable history defers cleanup. The existing
final-read/DELETE non-atomic race remains documented.
- The source documentation and generated lock metadata were updated with
gh-aw v1.2.2 and the existing pinned runtime.

## Validation and review

- 38 focused moderation safeguard tests pass, including the real
title-webhook shape, same-second cosmetic/title events, timestamp skew,
code whitespace, missing/deleted history, renewed requests, retry
exhaustion, and ambiguous DELETE completion.
- The regression controls reject the original merged helper.
- Required lint and TypeScript compilation pass; extension unit tests:
**3,573 passing, 9 pending**.
- Pinned workflow compilation/no-emit validation passes with zero
warnings.
- **19 actual GitHub Actions-token checks** against the final helper in
the private isolated fixture verify cosmetic edits retain the label,
real/combined/empty-description responses clear it, code corrections
count, and both central flags remain untouched. Examples:
[title-only](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44592069587),
[prose blank
line](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44592138593),
[combined
response](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44592295147),
[code
indentation](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44592539288),
[initially empty
report](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44591907981),
and [empty-to-filled
response](https://github.com/StellaHuang95/issue-moderation-postmerge-check/actions/runs/44593352325).
Fixture links require access; no production workflow was modified for
these checks.
- Independent Reviewer-agent review and follow-up walkthroughs addressed
the timestamp/history and code-prefix edge cases before final review.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code no-changelog review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants