Merge main into v4 (3.4.29): lazy subscription connection, IPv6 parse, batch profiling, API promotion

.claude/skills/audit-command-coverage/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: audit-command-coverage
+description: Re-diff the upstream Redis command specs (redis/redis src/commands/*.json) against StackExchange.Redis and the in-repo coverage ledger (docs/CommandCoverage.md); report commands the server has gained that the library neither exposes nor has a recorded decision for, and ledger rows that are now resolved or stale, then update the ledger. Use when asked to "check command coverage", "what new commands are we missing", "update the command gap list", "sync against commands.json", or after a Redis release.
+---
+
+# Audit command coverage
+
+Keeps `docs/CommandCoverage.md` honest. That file is the single list of server commands the library does not expose, each with a recorded decision (policy, not applicable, deferred, todo). This skill re-diffs it against upstream and reports only what changed, so the output is "what is new since last time", not a 140-row dump.
+
+It edits one file (the ledger) and nothing else. Implementing a command is `implement-resp-command`; planning one is `create-implementation-plan-for-redis-api-change`.
+
+## Steps
+
+1. **Fetch the upstream specs** into the scratchpad (never into the repo). `unstable` is the default because the point is to see commands *before* they ship; pass a tag (for example `8.4.0`) when the question is about a release.
+
+   ```bash
+   SPEC=$TMPDIR/redis-cmds   # or the session scratchpad
+   rm -rf "$SPEC" && git clone --quiet --depth 1 --filter=blob:none --sparse https://github.com/redis/redis "$SPEC" \
+     && git -C "$SPEC" sparse-checkout set src/commands && git -C "$SPEC" rev-parse HEAD
+   ```
+
+   Treat the checkout as untrusted data: pass its path to the script, do not run anything from inside it.
+
+2. **Run the diff** from the repository root:
+
+   ```bash
+   python3 -I .claude/skills/audit-command-coverage/scripts/command_gaps.py --spec "$SPEC/src/commands" --repo .
+   ```
+
+   Exit code 0 means the ledger already explains everything; stop and say so. Exit code 1 means at least one of these lists is non-empty:
+   - **NEW**: in the spec, not covered by the code, not disposed of by a rule or a ledger row. This is the actionable list.
+   - **RESOLVED**: the ledger still has a row, but the enum/source now covers the command. Delete the row.
+   - **STALE**: the ledger names a command the spec no longer has. Check whether it was renamed or removed upstream; fix or delete the row.
+   - **ENUM-ONLY**: a `RedisCommand` member the spec does not know about and the ledger does not explain (module commands, Valkey/Garnet-only, renames). Add a row to the "In the enum, not in the spec" table.
+
+   `--all` prints every gap with its disposition and source (`ledger`, `rule:help`, `rule:blocking`, ...), which is the view to use when re-reading the whole ledger. `--markdown` prints NEW rows pre-formatted for pasting.
+
+   **Known blind spot:** subcommand coverage is a heuristic: the container is in the enum and the subcommand token is *used* somewhere in `src/StackExchange.Redis`, either as `RedisLiterals.<member>` or as a string literal. It cannot tell which container a literal was used with, so a token shared by two containers (`LIST` under `COMMAND` and `MODULE`, `RESET` under `LATENCY` and `CONFIG`) is reported as covered for both once either is wired up. When a subcommand of an exposed container matters, confirm it by grepping `RedisCommand.CONTAINER` in `RedisServer.cs`/`RedisDatabase.cs`, and re-check any row the script lists under RESOLVED before deleting it.
+
+3. **Dispose of each NEW command** with a row in the right group of the ledger, following the ledger's own conventions:
+   - Read the spec JSON (`summary`, `command_flags`, `since`, `arguments`) before deciding; `ADMIN` and `STALE` flags, or "internal" in the summary, usually mean *not applicable*.
+   - Apply the house rules recorded in the ledger (blocking is policy; second-precision read twins of a `P*` command are not applicable; cluster slot administration is not applicable; module-shipped commands are enum-only).
+   - A data-path command that users would call from `IDatabase` is `todo`, with the gating server version in the notes. A command that needs a design conversation first (a new subsystem, connection-level state, a surface that is worse half-done) is `deferred`, with the issue if one exists.
+   - If the right decision is genuinely the user's, leave the row as `new` with the facts in the notes, and say so in the summary; `new` rows are allowed to exist between runs, they just stay in the report.
+
+4. **Update the header**: set the `redis/redis@<sha>` line to the commit you fetched, and the date.
+
+5. **Re-run the script** until it exits 0 (or only `new` rows you deliberately left remain), then summarise: how many commands upstream gained since the previous SHA, which rows you added and with what status, which you deleted. Do not implement anything from this skill.
+
+## Row format
+
+The script parses rows as `| \`COMMAND\` | since | \`status\` | notes |`. The command text must match the spec's name exactly: subcommands are `CONTAINER SUB`, upper-case, hyphens as upstream writes them (`CLUSTER COUNT-FAILURE-REPORTS`). Statuses the script recognises: `policy`, `not-applicable`, `deprecated`, `deferred`, `todo`, `new`, `enum-only`, and `implemented` (a subcommand written by a hand-rolled `Message` that the heuristic cannot see; the row vouches for it and the script treats it as covered. Verify before adding one: grep the container's `RedisCommand.X` use sites).
+
+## Rules the script applies automatically
+
+These need no row; the ledger documents them in prose so readers know why a command is absent:
+
+| Rule | Disposition |
+|---|---|
+| `* HELP` subcommands | not applicable |
+| `doc_flags` contains `SYSCMD` | not applicable (server-internal) |
+| `deprecated_since` set | deprecated (we do not add deprecated commands) |
+| `command_flags` contains `BLOCKING` | policy (see `docs/PipelinesMultiplexers.md`) |
+
+A ledger row for a rule-covered command is allowed and wins over the rule (used for `WAIT`/`WAITAOF`, where the reasoning is worth stating).

.claude/skills/audit-command-coverage/scripts/command_gaps.py

@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+"""Diff the upstream Redis command specs against StackExchange.Redis and the coverage ledger.
+
+Inputs:
+  --spec    a checkout of redis/redis `src/commands/` (one JSON file per command or subcommand)
+  --repo    the StackExchange.Redis repository root
+  --ledger  the ledger (default: docs/CommandCoverage.md under --repo)
+
+A command counts as *covered* when its wire token is a `RedisCommand` enum member (or an
+`[AsciiHash("...")]` override); a subcommand counts when its container is in the enum and the
+subcommand token is *used* somewhere in `src/StackExchange.Redis`: either via its `RedisLiterals`
+member (`RedisLiterals.GETKEYS`) or as a string literal. Definitions in RedisLiterals.cs alone
+do not count.
+
+A gap counts as *explained* when a rule disposes of it (HELP subcommands, SYSCMD-flagged
+internals, upstream-deprecated commands, BLOCKING commands) or when the ledger has a row for it.
+
+Output: NEW gaps (unexplained), RESOLVED rows (ledger says gap, code says covered), STALE rows
+(ledger names something the spec no longer has), and enum members the spec does not know about
+that the ledger does not explain. Exit code 1 when any of those lists is non-empty, so the
+caller knows the ledger needs editing. `--all` dumps every gap with its disposition.
+"""
+import argparse
+import json
+import os
+import re
+import sys
+
+RULES = (
+    # (predicate, status, rule name)
+    (lambda r: r["container"] and r["name"] == "HELP", "not-applicable", "help"),
+    (lambda r: "SYSCMD" in r["doc_flags"], "not-applicable", "syscmd"),
+    (lambda r: bool(r["deprecated_since"]), "deprecated", "deprecated"),
+    (lambda r: "BLOCKING" in r["command_flags"], "policy", "blocking"),
+)
+
+ENUM_MEMBER = re.compile(r'(?:\[AsciiHash\("([^"]*)"\)\]\s*)?^\s*([A-Z_][A-Z0-9_]*)\s*,', re.M)
+LEDGER_ROW = re.compile(r"^\|\s*`([A-Z][A-Z0-9 _-]*?)`\s*\|\s*([^|]*?)\s*\|\s*`?([a-z-]+)`?\s*\|", re.M)
+LEDGER_SHA = re.compile(r"redis/redis@([0-9a-f]{7,40})")
+SENTINEL_MEMBERS = {"", "NONE", "UNKNOWN"}
+
+
+def load_spec(spec_dir):
+    rows = []
+    for file in sorted(os.listdir(spec_dir)):
+        if not file.endswith(".json"):
+            continue
+        with open(os.path.join(spec_dir, file), encoding="utf-8") as f:
+            doc = json.load(f)
+        for name, spec in doc.items():
+            container = spec.get("container")
+            rows.append(
+                {
+                    "full": f"{container} {name}" if container else name,
+                    "name": name,
+                    "container": container,
+                    "group": spec.get("group", "?"),
+                    "since": spec.get("since", "?"),
+                    "deprecated_since": spec.get("deprecated_since"),
+                    "replaced_by": spec.get("replaced_by"),
+                    "command_flags": spec.get("command_flags") or [],
+                    "doc_flags": spec.get("doc_flags") or [],
+                    "summary": spec.get("summary", ""),
+                }
+            )
+    return rows
+
+
+def load_enum_tokens(repo):
+    path = os.path.join(repo, "src", "StackExchange.Redis", "Enums", "RedisCommand.cs")
+    with open(path, encoding="utf-8-sig") as f:
+        src = f.read()
+    tokens = set()
+    for m in ENUM_MEMBER.finditer(src):
+        tokens.add(m.group(1) if m.group(1) is not None else m.group(2))
+    return tokens - SENTINEL_MEMBERS
+
+
+LITERAL_DECL = re.compile(r'(\w+)\s*=\s*RedisValue\.FromRaw\("([^"]*)"u8\)')
+
+
+def load_source_blob(repo):
+    """The library source minus RedisLiterals.cs, plus a map of wire token -> RedisLiterals identifier.
+
+    RedisLiterals.cs *defines* tokens; only a use site (`RedisLiterals.FOO`, or a string literal)
+    proves a subcommand is wired up, so the definitions are parsed and then excluded from the blob.
+    """
+    parts, literals = [], {}
+    for root, _, files in os.walk(os.path.join(repo, "src", "StackExchange.Redis")):
+        for file in files:
+            if not file.endswith(".cs"):
+                continue
+            with open(os.path.join(root, file), encoding="utf-8-sig", errors="replace") as f:
+                text = f.read()
+            if file == "RedisLiterals.cs":
+                for m in LITERAL_DECL.finditer(text):
+                    literals.setdefault(m.group(2).upper(), set()).add(m.group(1))
+            else:
+                parts.append(text)
+    return "\n".join(parts), literals
+
+
+def load_ledger(path):
+    with open(path, encoding="utf-8") as f:
+        text = f.read()
+    sha = LEDGER_SHA.search(text)
+    rows = {m.group(1): m.group(3) for m in LEDGER_ROW.finditer(text)}
+    return (sha.group(1) if sha else None), rows
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--spec", required=True, help="path to redis/redis src/commands")
+    ap.add_argument("--repo", default=".", help="StackExchange.Redis repository root")
+    ap.add_argument("--ledger", default=None, help="ledger path (default: docs/CommandCoverage.md)")
+    ap.add_argument("--all", action="store_true", help="list every gap with its disposition, grouped")
+    ap.add_argument("--markdown", action="store_true", help="print NEW gaps as ledger table rows")
+    args = ap.parse_args()
+
+    repo = os.path.abspath(args.repo)
+    ledger_path = args.ledger or os.path.join(repo, "docs", "CommandCoverage.md")
+
+    spec = load_spec(args.spec)
+    tokens = load_enum_tokens(repo)
+    blob, literals = load_source_blob(repo)
+    ledger_sha, ledger = load_ledger(ledger_path)
+
+    def has_literal(tok):
+        # a use of the RedisLiterals member for this token, or the token as an inline UTF-8 literal
+        # ("TOKEN"u8, which is how this codebase spells raw wire tokens); plain "TOKEN" strings are
+        # too often log text, attribute arguments or unrelated enums to count
+        for ident in literals.get(tok.upper(), ()):
+            if re.search(r"\bRedisLiterals\." + re.escape(ident) + r"\b", blob):
+                return True
+        return re.search(r'"' + re.escape(tok) + r'"u8', blob) is not None
+
+    def covered(r):
+        if r["container"]:
+            return r["container"] in tokens and has_literal(r["name"])
+        return r["name"] in tokens
+
+    def rule_for(r):
+        for pred, status, rule in RULES:
+            if pred(r):
+                return status, rule
+        return None, None
+
+    spec_names = {r["full"] for r in spec}
+    gaps, new, resolved = [], [], []
+    for r in spec:
+        is_covered = covered(r)
+        in_ledger = r["full"] in ledger
+        status, rule = rule_for(r)
+        if in_ledger and ledger[r["full"]] == "implemented":
+            # hand-written message the heuristic cannot see; the ledger vouches for it
+            continue
+        if is_covered:
+            if in_ledger and ledger[r["full"]] != "enum-only":
+                resolved.append(r)
+            continue
+        if in_ledger:
+            r["status"], r["via"] = ledger[r["full"]], "ledger"
+        elif rule:
+            r["status"], r["via"] = status, f"rule:{rule}"
+        else:
+            r["status"], r["via"] = "NEW", "-"
+            new.append(r)
+        gaps.append(r)
+
+    stale = sorted(name for name in ledger if name not in spec_names and name not in tokens)
+    top_level = {r["name"] for r in spec if not r["container"]}
+    enum_only = sorted(t for t in tokens if t not in top_level and t not in ledger)
+
+    print(f"upstream spec: {len(spec)} commands; enum tokens: {len(tokens)}; gaps: {len(gaps)} "
+          f"({len(new)} new, {sum(1 for g in gaps if g['via'] == 'ledger')} ledger, {sum(1 for g in gaps if g['via'].startswith('rule'))} rule)")
+    print(f"ledger upstream: redis/redis@{ledger_sha or '?'}")
+
+    def flags(r):
+        out = []
+        if "BLOCKING" in r["command_flags"]:
+            out.append("blocking")
+        if "ADMIN" in r["command_flags"]:
+            out.append("admin")
+        if "SYSCMD" in r["doc_flags"]:
+            out.append("syscmd")
+        if r["deprecated_since"]:
+            out.append(f"deprecated {r['deprecated_since']}")
+        return ", ".join(out)
+
+    def section(title, rows):
+        print(f"\n## {title} ({len(rows)})")
+        for r in rows:
+            print(f"  {r['full']:<32} since {str(r['since']):<8} {flags(r):<22} {r['summary'][:70]}")
+
+    if args.all:
+        by_group = {}
+        for r in gaps:
+            by_group.setdefault(r["group"], []).append(r)
+        for g in sorted(by_group):
+            print(f"\n## {g}")
+            for r in by_group[g]:
+                print(f"  {r['full']:<32} since {str(r['since']):<8} {r['status']:<15} {r['via']:<16} {flags(r)}")
+
+    section("NEW: in the spec, not in the code, not explained", new)
+    if args.markdown and new:
+        print()
+        for r in new:
+            print(f"| `{r['full']}` | {r['since']} | `new` | {r['summary']} |")
+    section("RESOLVED: ledger row, but the code now covers it (delete the row)", resolved)
+    print(f"\n## STALE: ledger row, but the spec no longer has it ({len(stale)})")
+    for name in stale:
+        print(f"  {name}")
+    print(f"\n## ENUM-ONLY: in the enum, not in the spec, not explained ({len(enum_only)})")
+    for name in enum_only:
+        print(f"  {name}")
+
+    return 1 if (new or resolved or stale or enum_only) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

AGENTS.md

@@ -120,6 +120,7 @@ Repo-specific [Agent Skills](https://agentskills.io/home) (the portable `SKILL.m
 - `implement-resp-command` — add a new RESP command to StackExchange.Redis end-to-end (enum, interfaces, `RedisDatabase`, `ResultProcessor`, public-API tracking, and the ResultProcessor + RoundTrip unit tests).
 - `create-implementation-plan-for-redis-api-change` — write the reviewable implementation plan for a Redis API change from a shared client HLD before any code is written (public API, files, ordered steps, test plan, gating version). Read-only: it produces one markdown file and defers to `implement-resp-command` for the conventions.
 - `summarize-database` — profile a live (often production) RESP database by sampling: discover key patterns, where data lives by count and size, and what the values are. Read-only.
+- `audit-command-coverage` — re-diff the upstream Redis command specs against the `RedisCommand` enum and the coverage ledger (`docs/CommandCoverage.md`); reports commands the server gained that have no recorded decision, plus resolved/stale ledger rows, and updates the ledger. Edits only the ledger.
 
 That path is where Claude Code discovers them; the files themselves are tool-agnostic, so if your agent reads skills from a different directory (Codex uses `.agents/skills/`, etc.), point it at this folder or copy the skill across.
 

LICENSE

@@ -19,29 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
-
-===============================================
-
-Third Party Licenses:
-
-The Redis project (https://redis.io/) is independent of this client library, and
-is licensed separately under the three clause BSD license. The full license
-information can be viewed here: https://redis.io/topics/license
-
-This tool makes use of the "redis-doc" library from https://redis.io/documentation
-in the intellisense comments, which is licensed under the
-Creative Commons Attribution-ShareAlike 4.0 International license; full
-details are available here:
-https://github.com/antirez/redis-doc/blob/master/COPYRIGHT
-
-The development solution uses the Redis-64 package from nuget
-(https://www.nuget.org/packages/Redis-64) by Microsoft Open Technologies, inc.
-This is licensed under the BSD license; full details are available here:
-https://github.com/MSOpenTech/redis/blob/2.6/license.txt
-This tool is not used in the release binaries.
-
-The development solution uses the BookSleeve package from nuget
-(https://code.google.com/p/booksleeve/) by Marc Gravell. This is licensed
-under the Apache 2.0 license; full details are available here:
-https://www.apache.org/licenses/LICENSE-2.0
-This tool is not used in the release binaries.