Skip to content

Add KQL string matching rules to the coding guidelines - #2223

Open
RolandKrummenacher wants to merge 3 commits into
devfrom
docs/kql-coding-guidelines
Open

Add KQL string matching rules to the coding guidelines#2223
RolandKrummenacher wants to merge 3 commits into
devfrom
docs/kql-coding-guidelines

Conversation

@RolandKrummenacher

Copy link
Copy Markdown
Collaborator

🛠️ Description

The coding guidelines cover general practice, content, and the changelog, but say nothing about KQL — the primary language of the hub. docs-wiki/Coding-guidelines.md, AGENTS.md, and CONTRIBUTING.md don't mention KQL or Kusto at all today. That's part of why the redundant tolower() comparisons and word-search contains usages accumulated across the ingestion scripts until #2213 / #2220 — nothing in the project's written standards contradicted them.

This PR writes those rules down where both humans and AI assistants will find them.

docs-wiki/Coding-guidelines.md — a new ⚡ KQL section:

  • String comparison: why tolower() in comparison position is always wrong (every KQL string operator is already case-insensitive; the _cs forms are the case-sensitive ones), with an avoid/prefer table for the common rewrites — tolower(x) containshas, tolower(x) ===~, tolower(a) != tolower(b)!~, indexof(...) >= 0has, and dynamic operands not needing a tostring() wrapper.
  • When contains is required: framed as a semantic distinction (whole term vs. arbitrary substring) rather than a performance one, with the legitimate cases — needles fused inside a larger token (ConsumedUnit contains 'MB' also matching Mbps), word-stem matching, and fragments that never form a whole term. Also documents the term-index limits for punctuation and sub-3-character needles.
  • Verify before you swap: per-row cross-tabulation on real data rather than aggregate counts (which can hide offsetting false positives and negatives), plus a pointer to the executable equivalence harness added in Use case-insensitive KQL operators instead of tolower() comparisons #2220.

AGENTS.md — the rule is also stated inline under Coding Standards. That file already listed Bicep, PowerShell, markdown, and commit conventions but not KQL, and it's what both Claude Code (via CLAUDE.md) and Copilot (via .github/copilot-instructions.md) load. Stating it there means an assistant sees the rule without having to follow the link.

No behavior change — documentation only. The guidance is already enforced in CI by HubsKqlOperators.Tests.ps1 (from #2220); this PR documents the reasoning so contributors know why before the test tells them no.

📋 Checklist

🔬 How did you test this change?

  • 🤏 Lint tests — markdownlint-cli2 with the repo config: 0 issues

📦 Deploy to test?

Not applicable — documentation only.

🙋‍♀️ Do any of the following that apply?

  • 🚨 This is a breaking change.

📑 Did you update docs/changelog?

  • N/A — docs-wiki is contributor documentation, not part of the release changelog. It syncs to the GitHub wiki automatically on merge to dev.

🤖 Generated with Claude Code

The coding guidelines covered general practice, content, and the
changelog, but said nothing about KQL — the primary language of the
hub — so nothing contradicted the tolower()/contains patterns that
accumulated across the ingestion scripts until #2213/#2220.

Add a KQL section to docs-wiki/Coding-guidelines.md covering:
- Why tolower() in comparison position is always wrong (KQL string
  operators are already case-insensitive; _cs are the sensitive ones),
  with an avoid/prefer table for the common rewrites.
- has vs contains as a semantic distinction (whole term vs arbitrary
  substring), not just a performance one, including when contains is
  genuinely required and the term-index limits for short/punctuation
  needles.
- How to verify a swap: per-row cross-tabulation on real data rather
  than aggregate counts, plus the executable equivalence harness.

Also surface the rule directly in AGENTS.md so Claude Code and Copilot
see it without following the link — the file previously listed Bicep,
PowerShell, markdown and commit conventions, but not KQL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 31, 2026 07:26
@RolandKrummenacher
RolandKrummenacher requested review from a team, MSBrett and flanakin as code owners July 31, 2026 07:26
@microsoft-github-policy-service microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label Jul 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Documents project-specific KQL string-matching rules in the contributor coding standards so recurring anti-patterns (e.g., tolower() in comparisons and overuse of contains) are prevented via clear guidance alongside existing CI enforcement.

Changes:

  • Adds a new ⚡ KQL section to docs-wiki/Coding-guidelines.md covering case-insensitive operators, has vs contains, and validation guidance.
  • Adds a concise KQL rule summary to AGENTS.md under Coding Standards so assistants and contributors see it inline.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
docs-wiki/Coding-guidelines.md Introduces detailed KQL string matching guidance (operators, has vs contains, and verification steps).
AGENTS.md Adds an inline KQL standards summary referencing the coding guidelines and existing CI enforcement.
Suppressed comments (1)

docs-wiki/Coding-guidelines.md:65

  • This row states that =~ accepts a dynamic operand directly. That can be true for scalar dynamic values, but it is not universally safe if the field can be an object/array (or otherwise not string-like), and may produce confusing comparisons. Suggest qualifying the guidance to avoid implying tostring() is never needed for dynamic fields.
| `tostring(Dyn.Field) =~ 'true'` | `Dyn.Field =~ 'true'` | These operators accept a dynamic operand directly |

Comment thread docs-wiki/Coding-guidelines.md Outdated
Review flagged that 'these operators accept a dynamic operand directly'
was too absolute for fields that can hold an object or array.

Verified on a live cluster: tostring() is not the missing piece — raw
and tostring()-wrapped comparisons return identical results for every
payload shape, including objects and arrays. The real hazard is that a
non-scalar is compared against its JSON serialization, where has
matches a value nested anywhere in the text:

    dynamic({"nested":"true"}) has 'true'  -> true
    dynamic({"nested":"true"}) =~  'true'  -> false

Scope the table row to scalar dynamic values and add a note describing
the JSON-serialization behavior, why tostring() does not help, and what
to use instead (extract the member, or array_index_of/set_has_element
for arrays). All claims in the note executed against a cluster.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@RolandKrummenacher

Copy link
Copy Markdown
Collaborator Author

Good catch on the dynamic-operand row — it was too absolute. Qualified in 150c52d.

Verified the actual behavior on a live cluster first, and the result is a bit different from what the comment assumed: tostring() is not the missing piece. Raw and tostring()-wrapped comparisons return identical results for every payload shape, objects and arrays included:

d.v tostring(d.v) d.v =~ 'true' tostring(d.v) =~ 'true' d.v has 'true' tostring(d.v) has 'true'
"true" (string) true true true true true
true (bool) true true true true true
1 (number) 1 false false false false
{"nested":"true"} (object) {"nested":"true"} false false true true
["true"] (array) ["true"] false false true true
missing key (empty) false false false false

So the real hazard isn't a missing cast — it's that a non-scalar gets compared against its JSON serialization, where has matches a value nested anywhere in the text ({"nested":"true"} has 'true'true, while =~false). Adding tostring() produces the same JSON text and the same surprising result, so recommending it would give false confidence.

The table row is now scoped to scalar dynamic values, with an IMPORTANT note covering the JSON-serialization behavior, why tostring() doesn't help, and what to use instead — extract the member you mean (Dyn.Field.nested), or array_index_of() / set_has_element() for arrays. Every claim in that note was executed against a cluster.

Review flagged that the indexof(Col,'x') >= 0 -> Col has 'x' row was
not an equivalence: indexof() is substring-based while has is
term-based, so presenting it as a mechanical rewrite could lead
contributors into a behavior change. Verifying on a live cluster showed
the row was wrong on a second axis too — indexof() is case-sensitive:

  value            indexof>=0  contains  has
  'Windows Server'    true       true    true
  'WindowsServer'     true       true    false   <- term vs substring
  'MyWindowsBuild'    true       true    false   <- term vs substring
  'windows server'    false      true    false   <- case sensitivity

So indexof(Col,'x') >= 0 is exactly contains_cs, not contains and not
has. Reframe the row to keep the real advice (don't compute a position
you only need as a boolean) while stating both differences and telling
contributors to pick by intent. This mirrors the has-vs-contains
distinction the section already makes, which the original row
contradicted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs: Review 👀 PR that is ready to be reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants